[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:36 dist-test-slave-d0pf krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:36 dist-test-slave-d0pf krb5kdc[399](info): set up 2 sockets
Sep 24 17:38:36 dist-test-slave-d0pf krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 17:38:37.455533 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/test-user.keytab.
W20260924 17:38:37.692204 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.172s user 0.000s sys 0.001s
Sep 24 17:38:37 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 17:38:38.932904 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 1.216s user 0.002s sys 0.001s
I20260924 17:38:38.933434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:39.014425 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:39.014676 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:39.014701 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:39.015863 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:39.015906 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:39.015921 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:39.015933 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:39.015945 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:39.017357 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:39.017557 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:39.017768 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:39.022812 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:39.023231 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:39.023234 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:39.023379 412 server_base.cc:1034] running on GCE node
I20260924 17:38:39.023617 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:39.023973 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:39.025143 412 hybrid_clock.cc:648] HybridClock initialized: now 1790271519025127 us; error 34 us; skew 500 ppm
Sep 24 17:38:39 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:39.033354 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:39.033885 412 webserver.cc:460] Webserver started at http://127.0.98.62:43335/ using document root <none> and password file <none>
I20260924 17:38:39.034130 412 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:39.034178 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:39.034344 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:39.035554 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "d8613f86f82541b4920101e464e1c58e"
format_stamp: "Formatted at 2026-09-24 17:38:39 on dist-test-slave-d0pf"
server_key: "ab2df2258c2f0a73a137a159cdcbc46a"
server_key_iv: "706b679eeff8c23d5a13acfdcac08fb0"
server_key_version: "encryptionkey@0"
I20260924 17:38:39.035974 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8613f86f82541b4920101e464e1c58e"
format_stamp: "Formatted at 2026-09-24 17:38:39 on dist-test-slave-d0pf"
server_key: "ab2df2258c2f0a73a137a159cdcbc46a"
server_key_iv: "706b679eeff8c23d5a13acfdcac08fb0"
server_key_version: "encryptionkey@0"
I20260924 17:38:39.037393 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:39.038177 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:39.038363 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:39.038455 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "d8613f86f82541b4920101e464e1c58e"
format_stamp: "Formatted at 2026-09-24 17:38:39 on dist-test-slave-d0pf"
server_key: "ab2df2258c2f0a73a137a159cdcbc46a"
server_key_iv: "706b679eeff8c23d5a13acfdcac08fb0"
server_key_version: "encryptionkey@0"
I20260924 17:38:39.038528 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:39.059580 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:39.062034 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:39.062202 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:39.066228 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41359
I20260924 17:38:39.066262 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41359 every 8 connection(s)
I20260924 17:38:39.066635 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:39.067679 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:39.068933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 412
I20260924 17:38:39.069051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:39.069824 392 external_mini_cluster.cc:1428] Setting key 8107d80fa60520598b1d8b73e7e1ee40
I20260924 17:38:39.070878 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e: Bootstrap starting.
I20260924 17:38:39.071691 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:39.072011 480 log.cc:824] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:39.072798 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e: No bootstrap required, opened a new log
I20260924 17:38:39.075475 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } }
I20260924 17:38:39.075650 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:39.075690 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8613f86f82541b4920101e464e1c58e, State: Initialized, Role: FOLLOWER
I20260924 17:38:39.075825 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } }
I20260924 17:38:39.075911 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:39.075946 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:39.075994 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 0 FOLLOWER]: Advancing to term 1
Sep 24 17:38:39 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:39.086719 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:39.070784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.62:41359)
0924 17:38:39.070987 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:39.070998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:39.071251 (+ 253us) server_negotiation.cc:407] Connection header received
0924 17:38:39.074458 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:39.074466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:39.074542 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:39.074681 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:39.076963 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:39.077905 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:39.078976 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:39.079510 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:39.079612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:39.083136 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:39.083140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:39.083141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:39.083158 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:39.084924 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:39.085368 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:39.085371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:39.085371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:39.085464 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:39.085741 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:39.085742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:39.085743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:39.085989 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:39.086100 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:39.086337 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 17:38:39.086446 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260924 17:38:39.116765 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } }
I20260924 17:38:39.116973 480 leader_election.cc:302] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8613f86f82541b4920101e464e1c58e; no voters:
I20260924 17:38:39.117219 480 leader_election.cc:288] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:39.117328 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:39.117503 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:39.117547 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [term 1 LEADER]: Becoming Leader. State: Replica: d8613f86f82541b4920101e464e1c58e, State: Running, Role: LEADER
I20260924 17:38:39.117657 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } }
I20260924 17:38:39.118057 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8613f86f82541b4920101e464e1c58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } } }
I20260924 17:38:39.118144 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:39.118691 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8613f86f82541b4920101e464e1c58e. Latest consensus state: current_term: 1 leader_uuid: "d8613f86f82541b4920101e464e1c58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8613f86f82541b4920101e464e1c58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41359 } } }
I20260924 17:38:39.118746 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:39.118857 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:39.119496 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:39.121572 494 catalog_manager.cc:1347] Generated new cluster ID: addfbe49c48143e8ba17a39d7dcf8cee
I20260924 17:38:39.121624 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:38:39.147531 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:39.148113 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:39.150627 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8613f86f82541b4920101e464e1c58e: Generated new TSK 0
I20260924 17:38:39.150849 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 17:38:40.162159 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.007s user 0.000s sys 0.003s
I20260924 17:38:40.162557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:40.242427 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:40.242673 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:40.242714 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:40.243877 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:40.243920 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:40.243947 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:40.245303 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:40.245477 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:40.245646 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:38:40.250608 506 server_base.cc:1034] running on GCE node
W20260924 17:38:40.250574 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.250573 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.250617 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:40.251001 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:40.251318 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:40.252455 506 hybrid_clock.cc:648] HybridClock initialized: now 1790271520252445 us; error 30 us; skew 500 ppm
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:40.254914 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:40.255409 506 webserver.cc:460] Webserver started at http://127.0.98.1:33143/ using document root <none> and password file <none>
I20260924 17:38:40.255631 506 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:40.255676 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:40.255826 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:40.256872 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c265c0cc41c413fb97ca0e9a55cd617"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "56b9d1db683345840bb09d874f715dfa"
server_key_iv: "709da6db05135fa5b0ef394c66b4a0ae"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.257237 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c265c0cc41c413fb97ca0e9a55cd617"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "56b9d1db683345840bb09d874f715dfa"
server_key_iv: "709da6db05135fa5b0ef394c66b4a0ae"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.258543 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:38:40.259313 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.259479 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:40.259567 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "8c265c0cc41c413fb97ca0e9a55cd617"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "56b9d1db683345840bb09d874f715dfa"
server_key_iv: "709da6db05135fa5b0ef394c66b4a0ae"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.259636 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:40.276023 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:40.277971 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:40.278134 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:40.278400 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:40.278733 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:40.278790 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.278832 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:40.278859 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.284792 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44965
I20260924 17:38:40.284834 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44965 every 8 connection(s)
I20260924 17:38:40.285200 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:40.287093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 506
I20260924 17:38:40.287173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:40.287335 392 external_mini_cluster.cc:1428] Setting key 7c93fbf142196fae219ab7ad655b77d0
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:38:40.296344 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.285974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49941 (local address 127.0.98.62:41359)
0924 17:38:40.286086 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:40.286088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:40.286216 (+ 128us) server_negotiation.cc:407] Connection header received
0924 17:38:40.286694 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:40.286695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:40.286704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:40.286793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:40.288090 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.288839 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.290261 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.290451 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.290540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.293028 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:40.293029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:40.293030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:40.293036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:40.294543 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.294988 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.294989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.294989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.295108 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.295387 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.295388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.295388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.295508 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:40.295579 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:40.296142 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 17:38:40.296244 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
I20260924 17:38:40.296533 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.285936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41359 (local address 127.0.98.1:49941)
0924 17:38:40.286098 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:40.286140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:40.286616 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:40.286873 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:40.286876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:40.287697 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:40.287990 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.287997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.288924 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.288925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.290170 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.290172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.290546 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.290548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.290688 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.291285 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:40.291293 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:40.292926 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:40.294661 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.294668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.294670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.294894 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.295174 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.295175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.295176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.295316 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.295582 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:40.295589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:40.295926 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 17:38:40.296115 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 17:38:40.296295 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260924 17:38:40.297117 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41359
I20260924 17:38:40.297220 635 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:40.297479 635 heartbeater.cc:511] Master 127.0.98.62:41359 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 17:38:40.298138 444 ts_manager.cc:194] Registered new tserver with Master: 8c265c0cc41c413fb97ca0e9a55cd617 (127.0.98.1:44965)
I20260924 17:38:40.298759 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:40.313953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:40.396428 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:40.396652 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:40.396677 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:40.397797 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:40.397840 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:40.397867 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:40.399377 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:40.399566 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:40.399736 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:40.404675 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.404703 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.404778 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:40.404814 641 server_base.cc:1034] running on GCE node
I20260924 17:38:40.405047 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:40.405279 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:40.406457 641 hybrid_clock.cc:648] HybridClock initialized: now 1790271520406430 us; error 36 us; skew 500 ppm
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:40.408928 641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:40.409376 641 webserver.cc:460] Webserver started at http://127.0.98.2:34217/ using document root <none> and password file <none>
I20260924 17:38:40.409564 641 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:40.409603 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:40.409690 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:40.410671 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "d3eba3e0ea3a4d908daa6cb5d48a401b"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "c54de94f9bc57c7896d497a725eb9c54"
server_key_iv: "f3c8cfdacbf0583763b45b5588b0c1f5"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.410983 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d3eba3e0ea3a4d908daa6cb5d48a401b"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "c54de94f9bc57c7896d497a725eb9c54"
server_key_iv: "f3c8cfdacbf0583763b45b5588b0c1f5"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.412335 641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:40.413043 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.413226 641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.413329 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "d3eba3e0ea3a4d908daa6cb5d48a401b"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "c54de94f9bc57c7896d497a725eb9c54"
server_key_iv: "f3c8cfdacbf0583763b45b5588b0c1f5"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.413383 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:40.434733 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:40.436908 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:40.437031 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:40.437237 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:40.437580 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:40.437618 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.437649 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:40.437707 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.443915 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45583
I20260924 17:38:40.443970 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45583 every 8 connection(s)
I20260924 17:38:40.444303 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:40.448822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 641
I20260924 17:38:40.448930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:40.449115 392 external_mini_cluster.cc:1428] Setting key ef67c365b1ef5652bcfebd8d0fc1b67e
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:40.454159 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.445027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60803 (local address 127.0.98.62:41359)
0924 17:38:40.445109 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:40.445112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:40.445331 (+ 219us) server_negotiation.cc:407] Connection header received
0924 17:38:40.445800 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:40.445801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:40.445810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:40.445852 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:40.446795 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.447378 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.448348 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.448542 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.448597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.451213 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:40.451214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:40.451215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:40.451222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:40.452694 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.453148 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.453149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.453150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.453216 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.453484 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.453486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.453486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.453603 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:40.453677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:40.453973 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 17:38:40.454065 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:38:40.454356 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.445021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41359 (local address 127.0.98.2:60803)
0924 17:38:40.445181 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:40.445223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:40.445696 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:40.445911 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:40.445914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:40.446475 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:40.446667 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.446671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.447461 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.447461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.448264 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.448267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.448677 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.448678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.448829 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.449483 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:40.449494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:40.451103 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:40.452801 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.452808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.452811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.453050 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.453286 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.453287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.453288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.453412 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.453681 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:40.453687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:40.453841 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 17:38:40.453954 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 17:38:40.454118 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260924 17:38:40.454869 770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41359
I20260924 17:38:40.454963 770 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:40.455194 770 heartbeater.cc:511] Master 127.0.98.62:41359 requested a full tablet report, sending...
I20260924 17:38:40.455612 444 ts_manager.cc:194] Registered new tserver with Master: d3eba3e0ea3a4d908daa6cb5d48a401b (127.0.98.2:45583)
I20260924 17:38:40.456152 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60803
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:40.477550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:40.558899 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:40.559139 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:40.559165 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:40.560365 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:40.560422 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:40.560451 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:40.561960 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:40.562224 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:40.562475 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:40.567487 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.567504 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:40.567519 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:40.567718 775 server_base.cc:1034] running on GCE node
I20260924 17:38:40.567924 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:40.568166 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:40.569307 775 hybrid_clock.cc:648] HybridClock initialized: now 1790271520569263 us; error 60 us; skew 500 ppm
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:40.571769 775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:40.572248 775 webserver.cc:460] Webserver started at http://127.0.98.3:45403/ using document root <none> and password file <none>
I20260924 17:38:40.572475 775 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:40.572561 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:40.572705 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:40.573776 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "5715f948503f46739bf41ea981e0e007"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "b32792e4060f1640a130be79c0b4d47b"
server_key_iv: "ff078850312d8cfe6c91405d348269cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.574628 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5715f948503f46739bf41ea981e0e007"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "b32792e4060f1640a130be79c0b4d47b"
server_key_iv: "ff078850312d8cfe6c91405d348269cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.576099 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260924 17:38:40.576812 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.576987 775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:40.577085 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "5715f948503f46739bf41ea981e0e007"
format_stamp: "Formatted at 2026-09-24 17:38:40 on dist-test-slave-d0pf"
server_key: "b32792e4060f1640a130be79c0b4d47b"
server_key_iv: "ff078850312d8cfe6c91405d348269cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:40.577164 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:40.584370 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:40.586438 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:40.586608 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:40.586840 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:40.587224 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:40.587280 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.587323 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:40.587352 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:40.594347 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260924 17:38:40.594388 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
I20260924 17:38:40.594727 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:40.602294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 775
I20260924 17:38:40.602416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:40.602591 392 external_mini_cluster.cc:1428] Setting key 990db8ce2c253c6a8b1a9453ea9efe51
I20260924 17:38:40.604276 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.595461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38725 (local address 127.0.98.62:41359)
0924 17:38:40.595504 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:40.595506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:40.595736 (+ 230us) server_negotiation.cc:407] Connection header received
0924 17:38:40.596265 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:40.596266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:40.596278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:40.596341 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:40.597309 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.597810 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.598808 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.599015 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.599127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.601378 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:40.601379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:40.601379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:40.601402 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:40.602832 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.603299 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.603300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.603301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.603356 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.603616 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.603617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.603618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.603752 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:40.603814 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:40.604097 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 17:38:40.604177 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:38:40.604476 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.595437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41359 (local address 127.0.98.3:38725)
0924 17:38:40.595585 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:40.595623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:40.596148 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:40.596421 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:40.596425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:40.596989 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:40.597181 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.597185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.597905 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.597906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.598702 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:40.598704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.599135 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.599136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:40.599266 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.599816 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:40.599824 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:40.601282 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:40.602911 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.602918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.602920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.603225 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.603439 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:40.603439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:40.603440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:40.603548 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:40.603815 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:40.603821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:40.603965 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 17:38:40.604076 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:38:40.604236 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 17:38:40.604980 904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41359
I20260924 17:38:40.605078 904 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:40.605221 904 heartbeater.cc:511] Master 127.0.98.62:41359 requested a full tablet report, sending...
I20260924 17:38:40.605597 444 ts_manager.cc:194] Registered new tserver with Master: 5715f948503f46739bf41ea981e0e007 (127.0.98.3:42375)
I20260924 17:38:40.605988 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38725
I20260924 17:38:40.614620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:40.637491 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.629728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.1:44965)
0924 17:38:40.629898 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:40.629903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:40.629914 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:38:40.629940 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:40.629941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:40.629954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:40.630018 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:40.630547 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.631346 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.632038 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.632351 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.632405 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.634609 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:40.634618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:40.634618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:40.634626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:40.636274 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.636656 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.636660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.636660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.636722 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.636921 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.636923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.636923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.637073 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:40.637230 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:40.637287 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 17:38:40.637366 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
W20260924 17:38:40.637863 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36394
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:40.660388 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:40.653206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.98.1:44965)
0924 17:38:40.653276 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:40.653278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:40.653383 (+ 105us) server_negotiation.cc:407] Connection header received
0924 17:38:40.653473 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:40.653474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:40.653493 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:40.653554 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:40.654139 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.654630 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.655489 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:40.655695 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:40.655784 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:40.657951 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:40.657953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:40.657953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:40.657962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:40.659268 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.659625 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.659626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.659627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.659685 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:40.659880 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:40.659881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:40.659881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:40.659984 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:40.660110 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:40.660199 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:38:40.660282 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:38:40.660924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 506
I20260924 17:38:40.665989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 641
I20260924 17:38:40.671432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 775
I20260924 17:38:40.675989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 412
2026-09-24T17:38:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4276 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[918](info): set up 2 sockets
Sep 24 17:38:40 dist-test-slave-d0pf krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260924 17:38:41.718546 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:41 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:41.838900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:41.920081 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:41.920303 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:41.920326 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:41.921440 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:41.921485 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:41.921499 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:41.921512 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:41.921525 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:41.922930 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:41.923209 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:41.923403 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:41.928293 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:41.928368 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:41.928293 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:41.928511 931 server_base.cc:1034] running on GCE node
I20260924 17:38:41.928743 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:41.928998 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:41.930179 931 hybrid_clock.cc:648] HybridClock initialized: now 1790271521930166 us; error 25 us; skew 500 ppm
Sep 24 17:38:41 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:41.932789 931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:41.933231 931 webserver.cc:460] Webserver started at http://127.0.98.62:36303/ using document root <none> and password file <none>
I20260924 17:38:41.933465 931 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:41.933537 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:41.933699 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:41.934770 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1"
format_stamp: "Formatted at 2026-09-24 17:38:41 on dist-test-slave-d0pf"
server_key: "7e535491c2a01317f83442bdfcc72150"
server_key_iv: "f79d68cd3af8c6c53e19a909a2dd64ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:41.935191 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1"
format_stamp: "Formatted at 2026-09-24 17:38:41 on dist-test-slave-d0pf"
server_key: "7e535491c2a01317f83442bdfcc72150"
server_key_iv: "f79d68cd3af8c6c53e19a909a2dd64ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:41.936586 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:41.937379 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:41.937561 931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:41.937672 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1"
format_stamp: "Formatted at 2026-09-24 17:38:41 on dist-test-slave-d0pf"
server_key: "7e535491c2a01317f83442bdfcc72150"
server_key_iv: "f79d68cd3af8c6c53e19a909a2dd64ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:41.937748 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:41.949661 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:41.951721 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:41.951894 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:41.956002 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44813
I20260924 17:38:41.956035 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44813 every 8 connection(s)
I20260924 17:38:41.956401 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:41.957270 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:41.959542 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1: Bootstrap starting.
I20260924 17:38:41.960067 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:41.960307 999 log.cc:824] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:41.960922 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1: No bootstrap required, opened a new log
I20260924 17:38:41.962441 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260924 17:38:41.962579 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:41.962602 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b2ebaaecd45ef8b5c6609ab9121e1, State: Initialized, Role: FOLLOWER
I20260924 17:38:41.962706 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260924 17:38:41.962795 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:41.962828 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:41.962886 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:41.963495 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260924 17:38:41.963624 999 leader_election.cc:302] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9b2ebaaecd45ef8b5c6609ab9121e1; no voters:
I20260924 17:38:41.963797 999 leader_election.cc:288] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:41.963826 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:41.963905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 931
I20260924 17:38:41.963997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:41.964097 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:41.964195 392 external_mini_cluster.cc:1428] Setting key 54797ebbe88a393dd21e6897d6ed0b7a
I20260924 17:38:41.964150 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9b2ebaaecd45ef8b5c6609ab9121e1, State: Running, Role: LEADER
I20260924 17:38:41.964389 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260924 17:38:41.964744 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
I20260924 17:38:41.964963 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:41.964745 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9b2ebaaecd45ef8b5c6609ab9121e1. Latest consensus state: current_term: 1 leader_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9b2ebaaecd45ef8b5c6609ab9121e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
I20260924 17:38:41.965174 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:41.965520 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:41.966217 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:41.967811 1013 catalog_manager.cc:1347] Generated new cluster ID: 28f352cbb6794a28a9a4b85e8178f93e
I20260924 17:38:41.967866 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:38:41 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:41.975435 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:41.964707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.98.62:44813)
0924 17:38:41.964884 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:41.964973 (+ 89us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:41.965273 (+ 300us) server_negotiation.cc:407] Connection header received
0924 17:38:41.965453 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:41.965472 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:41.965539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:41.965696 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:41.966450 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:41.967811 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:41.968645 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:41.969077 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:41.969200 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:41.971574 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:41.971578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:41.971579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:41.971592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:41.973817 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:41.974213 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:41.974216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:41.974216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:41.974310 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:41.974529 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:41.974530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:41.974531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:41.974775 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:41.974985 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:41.975082 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 17:38:41.975227 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260924 17:38:41.989543 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:41.990149 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:41.993949 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d9b2ebaaecd45ef8b5c6609ab9121e1: Generated new TSK 0
I20260924 17:38:41.994175 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:42.033944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:42.140630 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:42.140901 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:42.140949 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:42.142185 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:42.142248 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:42.142294 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:42.143792 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:42.144053 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:42.144282 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:42.149392 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.149403 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.149436 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:42.149619 1025 server_base.cc:1034] running on GCE node
I20260924 17:38:42.149821 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:42.150025 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:42.151198 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790271522151186 us; error 43 us; skew 500 ppm
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:42.153673 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:42.154121 1025 webserver.cc:460] Webserver started at http://127.0.98.1:40509/ using document root <none> and password file <none>
I20260924 17:38:42.154349 1025 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:42.154421 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:42.154579 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:42.155746 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ab6ab054cb342449ba6dc990e354b23"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "ca3dc2f54d9ed81b3c797303ec11cdab"
server_key_iv: "b0c6aad9c9459b67f4d47a4888b5281c"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.156123 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ab6ab054cb342449ba6dc990e354b23"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "ca3dc2f54d9ed81b3c797303ec11cdab"
server_key_iv: "b0c6aad9c9459b67f4d47a4888b5281c"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.157476 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:42.158257 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.158458 1025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:42.158574 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "4ab6ab054cb342449ba6dc990e354b23"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "ca3dc2f54d9ed81b3c797303ec11cdab"
server_key_iv: "b0c6aad9c9459b67f4d47a4888b5281c"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.158664 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:42.187741 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:42.189937 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:42.190122 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:42.190372 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:42.190701 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:42.190757 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.190791 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:42.190819 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.196579 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35409
I20260924 17:38:42.196628 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35409 every 8 connection(s)
I20260924 17:38:42.196961 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:42.199807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1025
I20260924 17:38:42.199913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:42.200069 392 external_mini_cluster.cc:1428] Setting key e017e8df67b4f23116535929c63be781
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:42.207686 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.197667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43031 (local address 127.0.98.62:44813)
0924 17:38:42.197743 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.197746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.198024 (+ 278us) server_negotiation.cc:407] Connection header received
0924 17:38:42.198608 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.198609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.198623 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.198678 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.199840 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.200523 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.202018 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.202354 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.202434 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.204711 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.204713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.204713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.204722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.206221 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.206654 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.206655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.206656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.206717 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.206991 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.206992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.206992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.207151 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.207232 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.207512 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.207596 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:38:42.207896 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.197662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.1:43031)
0924 17:38:42.197847 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.197909 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.198492 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.198809 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.198813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.199513 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:42.199719 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.199725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.200624 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.200625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.201921 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.201924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.202485 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.202486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.202617 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.203236 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:42.203252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:42.204606 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:42.206323 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.206329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.206331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.206558 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.206783 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.206784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.206784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.206896 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.207220 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:42.207226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:42.207373 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.207490 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.207651 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:38:42.208443 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260924 17:38:42.208556 1154 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:42.208750 1154 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260924 17:38:42.209290 963 ts_manager.cc:194] Registered new tserver with Master: 4ab6ab054cb342449ba6dc990e354b23 (127.0.98.1:35409)
I20260924 17:38:42.209897 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:42.226478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:42.307992 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:42.308218 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:42.308243 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:42.309478 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:42.309530 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:42.309558 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:42.311177 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:42.311422 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:42.311687 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:42.316674 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.316704 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.316706 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:42.317077 1159 server_base.cc:1034] running on GCE node
I20260924 17:38:42.317299 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:42.317503 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:42.318686 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790271522318679 us; error 44 us; skew 500 ppm
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:42.321182 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:42.321623 1159 webserver.cc:460] Webserver started at http://127.0.98.2:44671/ using document root <none> and password file <none>
I20260924 17:38:42.321853 1159 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:42.321925 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:42.322083 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:42.323235 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "1400ab68cd144f2b8e3b94fdadc2b261"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "a0c9638924063360bd1d5900bec23da2"
server_key_iv: "0b10efe369233e43d0389b78d362d2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.323602 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1400ab68cd144f2b8e3b94fdadc2b261"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "a0c9638924063360bd1d5900bec23da2"
server_key_iv: "0b10efe369233e43d0389b78d362d2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.324977 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:42.325732 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.325915 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:42.326035 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "1400ab68cd144f2b8e3b94fdadc2b261"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "a0c9638924063360bd1d5900bec23da2"
server_key_iv: "0b10efe369233e43d0389b78d362d2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.326114 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:42.350486 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:42.352438 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:42.352617 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:42.352866 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:42.353193 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:42.353252 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.353296 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:42.353322 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.358979 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43353
I20260924 17:38:42.359093 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43353 every 8 connection(s)
I20260924 17:38:42.359401 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:42.361356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1159
I20260924 17:38:42.361457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:42.361652 392 external_mini_cluster.cc:1428] Setting key 8ae349a30e2c194a9737732a94e81788
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:42.370622 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.360112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40595 (local address 127.0.98.62:44813)
0924 17:38:42.360154 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.360157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.360416 (+ 259us) server_negotiation.cc:407] Connection header received
0924 17:38:42.360963 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.360965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.360977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.361042 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.362172 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.362801 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.364320 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.364701 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.364771 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.367696 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.367697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.367697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.367704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.369177 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.369664 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.369665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.369665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.369715 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.369981 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.369981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.369982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.370081 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.370133 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.370481 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.370551 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:38:42.370879 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.360093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.2:40595)
0924 17:38:42.360289 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.360324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.360844 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.361115 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.361119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.361836 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:42.362078 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.362083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.362894 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.362895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.364194 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.364197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.364793 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.364795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.364938 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.365518 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:42.365528 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:42.367594 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:42.369273 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.369280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.369282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.369583 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.369781 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.369782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.369783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.369915 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.370153 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:42.370159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:42.370334 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.370467 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.370652 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:38:42.371382 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260924 17:38:42.371490 1288 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:42.371690 1288 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260924 17:38:42.372092 963 ts_manager.cc:194] Registered new tserver with Master: 1400ab68cd144f2b8e3b94fdadc2b261 (127.0.98.2:43353)
I20260924 17:38:42.372480 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:42.390142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:42.470359 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:42.470602 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:42.470642 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:42.471889 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:42.471961 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:42.472011 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:42.473475 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:42.473763 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:42.473999 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:42.479084 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.479094 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:42.479106 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:42.479180 1293 server_base.cc:1034] running on GCE node
I20260924 17:38:42.479559 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:42.479826 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:42.480999 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790271522480993 us; error 36 us; skew 500 ppm
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:42.483799 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:42.484370 1293 webserver.cc:460] Webserver started at http://127.0.98.3:44429/ using document root <none> and password file <none>
I20260924 17:38:42.484575 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:42.484615 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:42.484712 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:42.485713 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "40c5fff45e5d4562a51598d50e2020c0"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "4160fe5a3ea21e7c0d53241f6de7d80f"
server_key_iv: "0ae1d85cca2d65986b699b1425b5f2e5"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.486003 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40c5fff45e5d4562a51598d50e2020c0"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "4160fe5a3ea21e7c0d53241f6de7d80f"
server_key_iv: "0ae1d85cca2d65986b699b1425b5f2e5"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.487300 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:42.488008 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.488233 1293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:42.488324 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "40c5fff45e5d4562a51598d50e2020c0"
format_stamp: "Formatted at 2026-09-24 17:38:42 on dist-test-slave-d0pf"
server_key: "4160fe5a3ea21e7c0d53241f6de7d80f"
server_key_iv: "0ae1d85cca2d65986b699b1425b5f2e5"
server_key_version: "encryptionkey@0"
I20260924 17:38:42.488395 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:42.498924 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:42.500753 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:42.500918 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:42.501152 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:42.501451 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:42.501484 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.501530 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:42.501559 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:42.507630 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41743
I20260924 17:38:42.507700 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41743 every 8 connection(s)
I20260924 17:38:42.508031 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:42.514775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1293
I20260924 17:38:42.514864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:42.515100 392 external_mini_cluster.cc:1428] Setting key 6b4ad4701488345627790e3547cdf225
I20260924 17:38:42.517076 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.508787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40233 (local address 127.0.98.62:44813)
0924 17:38:42.508845 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.508846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.509058 (+ 212us) server_negotiation.cc:407] Connection header received
0924 17:38:42.509511 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.509512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.509521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.509582 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.510460 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.510941 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.511994 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.512187 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.512242 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.514227 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.514229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.514229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.514236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.515870 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.516344 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.516345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.516345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.516397 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.516580 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.516580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.516580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.516684 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.516734 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.516939 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.516998 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:38:42.517242 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.508773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.3:40233)
0924 17:38:42.508917 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.508958 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.509433 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.509635 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.509638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.510189 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:42.510383 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.510388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.511028 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.511028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.511918 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.511920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.512261 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.512261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.512364 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.512811 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:42.512820 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:42.514137 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:42.516016 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.516024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.516026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.516280 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.516444 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:42.516444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:42.516445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:42.516530 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:42.516724 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:42.516726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:42.516834 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.516910 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.517028 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260924 17:38:42.517774 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260924 17:38:42.517891 1422 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:42.518072 1422 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260924 17:38:42.518462 963 ts_manager.cc:194] Registered new tserver with Master: 40c5fff45e5d4562a51598d50e2020c0 (127.0.98.3:41743)
I20260924 17:38:42.518838 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40233
I20260924 17:38:42.527158 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:38:42.534579 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.528641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.62:44813)
0924 17:38:42.528828 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.528830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.528887 (+ 57us) server_negotiation.cc:407] Connection header received
0924 17:38:42.528964 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.528964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.528974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.529035 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.529594 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.530128 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.530867 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.531129 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.531211 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.532308 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.532309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.532310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.532316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.533407 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.533775 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.533776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.533776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.533829 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.534050 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.534052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.534052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.534195 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.534307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.534402 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.534478 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 17:38:42.536567 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:42.538108 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:42.544776 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.540427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41743 (local address 127.0.0.1:43832)
0924 17:38:42.540575 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.540632 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.540701 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.540933 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.540935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.540937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:42.540997 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.540999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.542351 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.542352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.543618 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.543621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.544468 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.544553 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.544568 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.544638 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.544683 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260924 17:38:42.544793 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.540527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.98.3:41743)
0924 17:38:42.540694 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.540699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.540710 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:38:42.540786 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.540788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.540803 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.540889 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:42.541084 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.542262 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.543718 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.544378 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544453 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.544508 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.544675 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.544715 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260924 17:38:42.545357 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.541147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.0.98.1:35409)
0924 17:38:42.541395 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.541399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.541410 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:38:42.541448 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.541450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.541465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.541554 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:42.541750 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.542808 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544099 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.544846 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.544983 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.545221 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.545258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260924 17:38:42.545387 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.540870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35409 (local address 127.0.0.1:52090)
0924 17:38:42.540976 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.540986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.541024 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.541635 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.541635 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.541637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:42.541668 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.541669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.542908 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.542909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.544003 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.544005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.544933 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.544992 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.544996 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.545077 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.545320 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":76,"threads_started":1}
I20260924 17:38:42.545794 1356 tablet_service.cc:1467] Processing CreateTablet for tablet e7a4de08c136476d88e5d21d6a8ec7e7 (DEFAULT_TABLE table=test-table [id=74679a4bfdfb4da8a1e02ae36a9e04b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:42.546025 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.540683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43353 (local address 127.0.0.1:36284)
0924 17:38:42.540812 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.540822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.540866 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.542576 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.542577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.542579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:42.542619 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.542620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.543813 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.543814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.544716 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.544718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.545807 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.545808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.545869 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.545872 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.545925 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.545959 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":88,"threads_started":1}
I20260924 17:38:42.546042 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.541169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.98.2:43353)
0924 17:38:42.542426 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.542430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.542439 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:42.542473 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.542475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.542489 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.542560 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:42.542679 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.543715 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.544800 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.545739 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.545798 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.545846 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.545937 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.545972 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":94,"threads_started":1}
I20260924 17:38:42.546137 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a4de08c136476d88e5d21d6a8ec7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:42.546195 1088 tablet_service.cc:1467] Processing CreateTablet for tablet e7a4de08c136476d88e5d21d6a8ec7e7 (DEFAULT_TABLE table=test-table [id=74679a4bfdfb4da8a1e02ae36a9e04b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:42.546476 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a4de08c136476d88e5d21d6a8ec7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:42.546615 1222 tablet_service.cc:1467] Processing CreateTablet for tablet e7a4de08c136476d88e5d21d6a8ec7e7 (DEFAULT_TABLE table=test-table [id=74679a4bfdfb4da8a1e02ae36a9e04b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:42.546852 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a4de08c136476d88e5d21d6a8ec7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:42.549511 1439 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Bootstrap starting.
I20260924 17:38:42.549933 1441 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Bootstrap starting.
I20260924 17:38:42.550179 1439 tablet_bootstrap.cc:654] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:42.550601 1441 tablet_bootstrap.cc:654] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:42.550758 1439 log.cc:824] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:42.551192 1440 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Bootstrap starting.
I20260924 17:38:42.551203 1441 log.cc:824] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:42.551606 1439 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: No bootstrap required, opened a new log
I20260924 17:38:42.551707 1439 ts_tablet_manager.cc:1397] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:42.551812 1440 tablet_bootstrap.cc:654] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:42.552122 1441 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: No bootstrap required, opened a new log
I20260924 17:38:42.552206 1441 ts_tablet_manager.cc:1397] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:42.552420 1440 log.cc:824] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:42.553100 1440 tablet_bootstrap.cc:492] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: No bootstrap required, opened a new log
I20260924 17:38:42.553182 1440 ts_tablet_manager.cc:1397] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:42.553700 1439 raft_consensus.cc:348] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.553897 1439 raft_consensus.cc:374] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:42.553936 1439 raft_consensus.cc:729] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1400ab68cd144f2b8e3b94fdadc2b261, State: Initialized, Role: FOLLOWER
I20260924 17:38:42.554095 1439 consensus_queue.cc:260] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.554098 1441 raft_consensus.cc:348] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.554283 1441 raft_consensus.cc:374] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:42.554329 1441 raft_consensus.cc:729] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab6ab054cb342449ba6dc990e354b23, State: Initialized, Role: FOLLOWER
I20260924 17:38:42.554376 1439 ts_tablet_manager.cc:1428] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:38:42.554445 1441 consensus_queue.cc:260] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.554540 1288 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
I20260924 17:38:42.554708 1441 ts_tablet_manager.cc:1428] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:38:42.554713 1154 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
I20260924 17:38:42.554925 1440 raft_consensus.cc:348] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.555126 1440 raft_consensus.cc:374] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:42.555176 1440 raft_consensus.cc:729] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c5fff45e5d4562a51598d50e2020c0, State: Initialized, Role: FOLLOWER
I20260924 17:38:42.555289 1440 consensus_queue.cc:260] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.555545 1440 ts_tablet_manager.cc:1428] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 17:38:42.555589 1422 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
W20260924 17:38:42.610001 1289 tablet.cc:2369] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:38:42.697782 1155 tablet.cc:2369] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:42.732372 1447 raft_consensus.cc:482] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:42.732538 1447 raft_consensus.cc:504] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.732914 1447 leader_election.cc:288] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1400ab68cd144f2b8e3b94fdadc2b261 (127.0.98.2:43353), 4ab6ab054cb342449ba6dc990e354b23 (127.0.98.1:35409)
I20260924 17:38:42.736050 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.733319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42161 (local address 127.0.98.2:43353)
0924 17:38:42.733387 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.733389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.733404 (+ 15us) server_negotiation.cc:407] Connection header received
0924 17:38:42.733436 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.733437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.733449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.733559 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:42.733719 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.734193 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735248 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.735645 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735770 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.735782 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.735944 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.735986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:38:42.736050 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.733195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43353 (local address 127.0.98.3:42161)
0924 17:38:42.733268 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.733281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.733376 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.733566 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.733567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.733569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:42.733640 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.733642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.734293 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.734294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.735157 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.735160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.735767 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.735858 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.735861 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.735926 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.735970 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 17:38:42.736207 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.733330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35409 (local address 127.0.98.3:52455)
0924 17:38:42.733497 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:42.733506 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:42.733580 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:42.733744 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:42.733744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:42.733746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:42.733817 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.733823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.734513 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.734514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.735383 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:42.735385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.735923 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:42.736006 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.736009 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:42.736086 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 17:38:42.736125 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":11,"thread_start_us":104,"threads_started":1}
I20260924 17:38:42.736198 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.733410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52455 (local address 127.0.98.1:35409)
0924 17:38:42.733469 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.733471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.733570 (+ 99us) server_negotiation.cc:407] Connection header received
0924 17:38:42.733635 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.733636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.733653 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.733738 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:42.733895 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.734409 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735477 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.735825 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.735921 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.735930 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.736095 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.736134 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:38:42.736475 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a4de08c136476d88e5d21d6a8ec7e7" candidate_uuid: "40c5fff45e5d4562a51598d50e2020c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" is_pre_election: true
I20260924 17:38:42.736516 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a4de08c136476d88e5d21d6a8ec7e7" candidate_uuid: "40c5fff45e5d4562a51598d50e2020c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab6ab054cb342449ba6dc990e354b23" is_pre_election: true
I20260924 17:38:42.736655 1108 raft_consensus.cc:2454] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c5fff45e5d4562a51598d50e2020c0 in term 0.
I20260924 17:38:42.736655 1242 raft_consensus.cc:2454] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c5fff45e5d4562a51598d50e2020c0 in term 0.
I20260924 17:38:42.736874 1311 leader_election.cc:302] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1400ab68cd144f2b8e3b94fdadc2b261, 40c5fff45e5d4562a51598d50e2020c0; no voters:
I20260924 17:38:42.737066 1447 raft_consensus.cc:2781] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:42.737131 1447 raft_consensus.cc:482] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:42.737149 1447 raft_consensus.cc:3037] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:42.737890 1447 raft_consensus.cc:504] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.738000 1447 leader_election.cc:288] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1400ab68cd144f2b8e3b94fdadc2b261 (127.0.98.2:43353), 4ab6ab054cb342449ba6dc990e354b23 (127.0.98.1:35409)
I20260924 17:38:42.738215 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a4de08c136476d88e5d21d6a8ec7e7" candidate_uuid: "40c5fff45e5d4562a51598d50e2020c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1400ab68cd144f2b8e3b94fdadc2b261"
I20260924 17:38:42.738252 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a4de08c136476d88e5d21d6a8ec7e7" candidate_uuid: "40c5fff45e5d4562a51598d50e2020c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab6ab054cb342449ba6dc990e354b23"
I20260924 17:38:42.738307 1242 raft_consensus.cc:3037] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:42.738318 1108 raft_consensus.cc:3037] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:42.739271 1242 raft_consensus.cc:2454] T e7a4de08c136476d88e5d21d6a8ec7e7 P 1400ab68cd144f2b8e3b94fdadc2b261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c5fff45e5d4562a51598d50e2020c0 in term 1.
I20260924 17:38:42.739271 1108 raft_consensus.cc:2454] T e7a4de08c136476d88e5d21d6a8ec7e7 P 4ab6ab054cb342449ba6dc990e354b23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c5fff45e5d4562a51598d50e2020c0 in term 1.
I20260924 17:38:42.739470 1311 leader_election.cc:302] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c5fff45e5d4562a51598d50e2020c0, 4ab6ab054cb342449ba6dc990e354b23; no voters:
I20260924 17:38:42.739578 1447 raft_consensus.cc:2781] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:42.739724 1447 raft_consensus.cc:686] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [term 1 LEADER]: Becoming Leader. State: Replica: 40c5fff45e5d4562a51598d50e2020c0, State: Running, Role: LEADER
I20260924 17:38:42.739830 1447 consensus_queue.cc:237] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } }
I20260924 17:38:42.740777 962 catalog_manager.cc:5495] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c5fff45e5d4562a51598d50e2020c0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "40c5fff45e5d4562a51598d50e2020c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c5fff45e5d4562a51598d50e2020c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1400ab68cd144f2b8e3b94fdadc2b261" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab6ab054cb342449ba6dc990e354b23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35409 } health_report { overall_health: UNKNOWN } } }
W20260924 17:38:42.758600 1423 tablet.cc:2369] T e7a4de08c136476d88e5d21d6a8ec7e7 P 40c5fff45e5d4562a51598d50e2020c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:42.774628 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.767500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.98.1:35409)
0924 17:38:42.767553 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.767565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.767646 (+ 81us) server_negotiation.cc:407] Connection header received
0924 17:38:42.767699 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.767700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.767710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.767758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.768302 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.768780 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.769452 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.769644 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.769700 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.772009 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.772021 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.772021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.772028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.773683 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.773977 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.773980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.773980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.774038 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.774182 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.774182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.774182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.774300 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.774401 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.774451 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.774509 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:42.797690 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.790315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.98.1:35409)
0924 17:38:42.790407 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.790409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.790460 (+ 51us) server_negotiation.cc:407] Connection header received
0924 17:38:42.790520 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.790521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.790530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.790580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.791191 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.791768 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.792471 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.792651 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.792714 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.795263 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.795264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.795265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.795272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.796470 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.796830 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.796831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.796832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.796894 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.797096 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.797096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.797097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.797201 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.797306 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.797496 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.797578 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:42.819427 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:42.812571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.1:35409)
0924 17:38:42.812640 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:42.812642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:42.812704 (+ 62us) server_negotiation.cc:407] Connection header received
0924 17:38:42.812753 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:42.812753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:42.812763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:42.812814 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:42.813395 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.813862 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.814582 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:42.814761 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:42.814810 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:42.817095 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:42.817096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:42.817097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:42.817104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:42.818343 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.818689 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.818690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.818690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.818738 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:42.818902 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:42.818902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:42.818903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:42.818991 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:42.819143 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:42.819262 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 17:38:42.819325 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 17:38:42.819944 1068 tablet_service.cc:2908] Rejecting scan request for tablet e7a4de08c136476d88e5d21d6a8ec7e7: Uninitialized: clean time has not yet been initialized
I20260924 17:38:42.821218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1025
I20260924 17:38:42.826084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1159
I20260924 17:38:42.830662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1293
I20260924 17:38:42.835297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 931
2026-09-24T17:38:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2160 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[1463](info): set up 2 sockets
Sep 24 17:38:42 dist-test-slave-d0pf krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260924 17:38:43.914731 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:43 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:44.034653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:44.116164 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:44.116448 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:44.116488 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:44.117686 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:44.117775 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:38:44.117798 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:44.117820 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:44.117838 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:44.117866 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:44.119647 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35603
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:44.119953 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:44.120193 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:44.125259 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.125273 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.125459 1476 server_base.cc:1034] running on GCE node
W20260924 17:38:44.125243 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.125790 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:44.126122 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:44.127276 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790271524127249 us; error 46 us; skew 500 ppm
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:44.133526 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:44.134013 1476 webserver.cc:460] Webserver started at http://127.0.98.62:34063/ using document root <none> and password file <none>
I20260924 17:38:44.134277 1476 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:44.134326 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:44.134488 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:44.135691 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "341a27f0d59c478798850d31a9418616"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "e59d5258933d58c1dfc3d5ec0c0655f3"
server_key_iv: "21e02a98a87eb358b42114fe3756cf01"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.136068 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "341a27f0d59c478798850d31a9418616"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "e59d5258933d58c1dfc3d5ec0c0655f3"
server_key_iv: "21e02a98a87eb358b42114fe3756cf01"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.137439 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:44.138151 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.138350 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:44.138433 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "341a27f0d59c478798850d31a9418616"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "e59d5258933d58c1dfc3d5ec0c0655f3"
server_key_iv: "21e02a98a87eb358b42114fe3756cf01"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.138525 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:44.186755 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:44.189201 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:44.189359 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:44.193964 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35603
I20260924 17:38:44.194046 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35603 every 8 connection(s)
I20260924 17:38:44.194396 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:44.195082 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.197386 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Bootstrap starting.
I20260924 17:38:44.198010 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.198314 1554 log.cc:824] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:44.199306 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: No bootstrap required, opened a new log
I20260924 17:38:44.200181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1476
I20260924 17:38:44.200281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:44.200443 392 external_mini_cluster.cc:1428] Setting key cfb77872b91772ebf5e9ffc6262c7fd9
I20260924 17:38:44.201921 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } }
I20260924 17:38:44.202104 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.202147 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341a27f0d59c478798850d31a9418616, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.202298 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } }
I20260924 17:38:44.202391 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:44.202425 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:44.202478 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.203226 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } }
I20260924 17:38:44.203356 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 341a27f0d59c478798850d31a9418616; no voters:
I20260924 17:38:44.203528 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:44.203583 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:44.203720 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [term 1 LEADER]: Becoming Leader. State: Replica: 341a27f0d59c478798850d31a9418616, State: Running, Role: LEADER
I20260924 17:38:44.203789 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:44.203871 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } }
I20260924 17:38:44.204181 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 341a27f0d59c478798850d31a9418616. Latest consensus state: current_term: 1 leader_uuid: "341a27f0d59c478798850d31a9418616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } } }
I20260924 17:38:44.204286 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:44.204195 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "341a27f0d59c478798850d31a9418616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341a27f0d59c478798850d31a9418616" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35603 } } }
I20260924 17:38:44.204463 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616 [sys.catalog]: This master's current role is: LEADER
W20260924 17:38:44.205691 1575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:38:44.205760 1575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:38:44.205910 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:44.206100 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:44.207824 1566 catalog_manager.cc:1347] Generated new cluster ID: 4d786124034f4e519588ea475eeadcd5
I20260924 17:38:44.207881 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.213285 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.200974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.0.98.62:35603)
0924 17:38:44.201156 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.201172 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.201197 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:38:44.201899 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.201906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.201954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.202071 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.202880 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.204230 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.205289 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.205928 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.206063 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.208853 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.208857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.208857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.208870 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.211486 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.211884 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.211887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.211888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.212015 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.212228 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.212229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.212230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.212544 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.212716 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.212809 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.213062 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260924 17:38:44.223866 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:44.224622 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:44.230450 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Generated new TSK 0
I20260924 17:38:44.230654 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:44.258378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35603
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:44.339821 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:44.340031 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:44.340059 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:44.341382 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:44.341424 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:44.341445 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:44.341499 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:44.343191 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:44.343429 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:44.343672 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:44.348829 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.348858 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.348856 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.348840 1580 server_base.cc:1034] running on GCE node
I20260924 17:38:44.349227 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:44.349483 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:44.350636 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790271524350621 us; error 29 us; skew 500 ppm
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:44.353295 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:44.353781 1580 webserver.cc:460] Webserver started at http://127.0.98.1:45833/ using document root <none> and password file <none>
I20260924 17:38:44.353966 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:44.354007 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:44.354096 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:44.355247 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "d150c488c37e4db79088d8be184f3637"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "43e52e1066edf1ddb4d9b18ec5eb1d77"
server_key_iv: "a4f26d04c5925e3523aef626fb899ec4"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.355568 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d150c488c37e4db79088d8be184f3637"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "43e52e1066edf1ddb4d9b18ec5eb1d77"
server_key_iv: "a4f26d04c5925e3523aef626fb899ec4"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.357000 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:38:44.357767 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.357940 1580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:44.358047 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "d150c488c37e4db79088d8be184f3637"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "43e52e1066edf1ddb4d9b18ec5eb1d77"
server_key_iv: "a4f26d04c5925e3523aef626fb899ec4"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.358125 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:44.374967 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:44.377529 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:44.377710 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:44.378324 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:44.378388 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.378458 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:44.378489 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.385763 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45517
I20260924 17:38:44.385844 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45517 every 8 connection(s)
I20260924 17:38:44.386270 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:44.388638 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.378792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55091 (local address 127.0.98.62:35603)
0924 17:38:44.378867 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.378870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.379132 (+ 262us) server_negotiation.cc:407] Connection header received
0924 17:38:44.379683 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.379685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.379703 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.379785 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.380523 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.381374 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.382890 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.383376 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.383471 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.385835 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.385837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.385837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.385846 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.387400 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.387768 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.387768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.387769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.387821 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.388066 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.388067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.388068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.388214 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.388279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.388502 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.388574 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:38:44.388856 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.378796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.1:55091)
0924 17:38:44.378969 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.379010 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.379571 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.379924 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.379929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.380244 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.380442 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.380447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.381474 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.381475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.382791 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.382795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.383490 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.383491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.383638 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.384225 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.384233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.385718 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.387471 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.387476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.387477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.387695 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.387878 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.387878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.387879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.387973 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.388281 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.388285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.388400 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.388481 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.388601 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260924 17:38:44.389391 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35603
I20260924 17:38:44.389475 1711 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:44.389663 1711 heartbeater.cc:511] Master 127.0.98.62:35603 requested a full tablet report, sending...
I20260924 17:38:44.390206 1507 ts_manager.cc:194] Registered new tserver with Master: d150c488c37e4db79088d8be184f3637 (127.0.98.1:45517)
I20260924 17:38:44.390836 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55091
I20260924 17:38:44.393337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1580
I20260924 17:38:44.393438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:44.393608 392 external_mini_cluster.cc:1428] Setting key 69cf043a4cc7dbf79ef39ba4efc1375d
I20260924 17:38:44.397536 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.391211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35303 (local address 127.0.98.62:35603)
0924 17:38:44.391253 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.391255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.391377 (+ 122us) server_negotiation.cc:407] Connection header received
0924 17:38:44.391449 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.391450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.391460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.391515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.391835 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.392336 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.393050 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.393356 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.393435 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.394647 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.394649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.394649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.394657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.396369 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.396784 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.396785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.396786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.396833 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.397031 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.397032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.397032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.397124 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.397204 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.397366 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.397443 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:38:44.397555 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.391153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.1:35303)
0924 17:38:44.391313 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.391322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.391392 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.391520 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.391521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.391694 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.391752 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.391754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.392432 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.392433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.392957 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.392958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.393457 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.393459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.393534 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.394017 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.394022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.394542 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.396456 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.396457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.396458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.396708 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.396897 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.396898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.396898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.396969 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.397199 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.397200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.397247 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.397341 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.397444 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:44.422380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35603
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:44.504364 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:44.504639 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:44.504667 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:44.505895 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:44.505936 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:44.505957 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:44.505975 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:44.507517 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:44.507706 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:44.507879 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:44.512837 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.512880 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.512940 1720 server_base.cc:1034] running on GCE node
W20260924 17:38:44.512892 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.513231 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:44.513513 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:44.514706 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790271524514692 us; error 25 us; skew 500 ppm
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:44.517155 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:44.517634 1720 webserver.cc:460] Webserver started at http://127.0.98.2:40545/ using document root <none> and password file <none>
I20260924 17:38:44.517858 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:44.517928 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:44.518077 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:44.519196 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "5193e20be1764757bced8f6c54874ec4"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "221a6fdf55655537c131e88a95b225bd"
server_key_iv: "823e7eace161eb0eb2f8c68548d5f35b"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.519548 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5193e20be1764757bced8f6c54874ec4"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "221a6fdf55655537c131e88a95b225bd"
server_key_iv: "823e7eace161eb0eb2f8c68548d5f35b"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.520862 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:44.522195 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.522397 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:44.522475 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "5193e20be1764757bced8f6c54874ec4"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "221a6fdf55655537c131e88a95b225bd"
server_key_iv: "823e7eace161eb0eb2f8c68548d5f35b"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.522529 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:44.535733 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:44.537540 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:44.537678 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:44.538165 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:44.538239 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.538286 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:44.538314 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.545719 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42857
I20260924 17:38:44.545809 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42857 every 8 connection(s)
I20260924 17:38:44.546066 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:44.547251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1720
I20260924 17:38:44.547362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:44.547551 392 external_mini_cluster.cc:1428] Setting key 083045f57f4f7f1deb1bc2a0bf980f97
I20260924 17:38:44.548913 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.538846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40039 (local address 127.0.98.62:35603)
0924 17:38:44.538911 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.538915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.539202 (+ 287us) server_negotiation.cc:407] Connection header received
0924 17:38:44.539738 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.539740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.539752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.539794 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.540485 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.541133 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.542460 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.542778 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.542851 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.545559 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.545561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.545561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.545571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.547318 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.547873 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.547875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.547876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.547947 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.548241 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.548242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.548242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.548370 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.548453 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.548777 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.548848 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:38:44.549227 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.538855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.2:40039)
0924 17:38:44.539026 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.539109 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.539626 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.539877 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.539881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.540188 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.540385 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.540390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.541230 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.541231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.542356 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.542359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.542877 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.542878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.543023 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.543633 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.543645 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.545437 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.547436 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.547442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.547444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.547753 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.548028 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.548029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.548029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.548141 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.548444 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.548450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.548617 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.548742 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.548934 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260924 17:38:44.549777 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35603
I20260924 17:38:44.549871 1851 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:44.550076 1851 heartbeater.cc:511] Master 127.0.98.62:35603 requested a full tablet report, sending...
I20260924 17:38:44.550434 1507 ts_manager.cc:194] Registered new tserver with Master: 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857)
I20260924 17:38:44.550788 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40039
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:38:44.557061 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.551000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58877 (local address 127.0.98.62:35603)
0924 17:38:44.551065 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.551068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.551220 (+ 152us) server_negotiation.cc:407] Connection header received
0924 17:38:44.551296 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.551297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.551307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.551361 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.551692 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.552174 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.552912 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.553113 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.553200 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.554396 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.554398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.554398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.554406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.555845 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.556217 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.556218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.556218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.556302 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.556502 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.556503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.556503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.556646 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.556729 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.556890 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.556966 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:38:44.557075 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.551009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.2:58877)
0924 17:38:44.551161 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.551174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.551234 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.551366 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.551367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.551550 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.551598 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.551599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.552272 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.552273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.552807 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.552809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.553202 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.553203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.553272 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.553914 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.553919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.554294 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.555935 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.555937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.555938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.556109 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.556371 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.556372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.556372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.556452 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.556729 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.556730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.556776 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.556865 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.556968 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:44.574842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35603
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:44.658129 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:44.658372 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:44.658411 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:44.659641 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:44.659708 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:44.659742 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:44.659788 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:44.661371 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:44.661643 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:44.661887 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:44.666898 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.666906 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:44.667080 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:44.667313 1860 server_base.cc:1034] running on GCE node
I20260924 17:38:44.667516 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:44.667776 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:44.668960 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790271524668943 us; error 32 us; skew 500 ppm
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:44.671573 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:44.672049 1860 webserver.cc:460] Webserver started at http://127.0.98.3:37617/ using document root <none> and password file <none>
I20260924 17:38:44.672315 1860 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:44.672392 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:44.672552 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:44.673631 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "b396ae2783ae756873e5948695a4d057"
server_key_iv: "e6cbb6cb5c1ee8540121b969491131ac"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.674014 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "b396ae2783ae756873e5948695a4d057"
server_key_iv: "e6cbb6cb5c1ee8540121b969491131ac"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.675441 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:38:44.676244 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.676451 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:44.676564 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
format_stamp: "Formatted at 2026-09-24 17:38:44 on dist-test-slave-d0pf"
server_key: "b396ae2783ae756873e5948695a4d057"
server_key_iv: "e6cbb6cb5c1ee8540121b969491131ac"
server_key_version: "encryptionkey@0"
I20260924 17:38:44.676641 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:44.721562 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:44.724777 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:44.724969 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:44.725638 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:44.725700 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.725732 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:44.725749 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.733417 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45053
I20260924 17:38:44.733494 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45053 every 8 connection(s)
I20260924 17:38:44.733824 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:38:44.736490 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.726075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37285 (local address 127.0.98.62:35603)
0924 17:38:44.726149 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.726152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.726366 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:38:44.727129 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.727131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.727144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.727199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.728088 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.728943 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.730662 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.730991 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.731115 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.733663 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.733665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.733665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.733672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.735290 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.735685 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.735686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.735687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.735741 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.735956 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.735956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.735956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.736062 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.736111 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.736364 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.736425 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:38:44.736636 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.726087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.3:37285)
0924 17:38:44.726248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.726289 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.726974 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.727276 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.727281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.727664 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.727970 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.727978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.729073 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.729075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.730550 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.730554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.731135 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.731136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.731376 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.731962 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.731971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.733528 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.735390 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.735395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.735396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.735613 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.735798 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.735799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.735799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.735900 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.736115 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.736119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.736237 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.736320 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.736442 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260924 17:38:44.737107 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35603
I20260924 17:38:44.737188 1991 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:44.737394 1991 heartbeater.cc:511] Master 127.0.98.62:35603 requested a full tablet report, sending...
I20260924 17:38:44.737813 1508 ts_manager.cc:194] Registered new tserver with Master: 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:44.738317 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37285
I20260924 17:38:44.740106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 1860
I20260924 17:38:44.740219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:44.740445 392 external_mini_cluster.cc:1428] Setting key 99bc840da9845f4259cfbeacbf8efa7d
I20260924 17:38:44.741217 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:38:44.744642 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.738239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57081 (local address 127.0.98.62:35603)
0924 17:38:44.738281 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.738283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.738385 (+ 102us) server_negotiation.cc:407] Connection header received
0924 17:38:44.738481 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.738482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.738493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.738548 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.738878 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.739540 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.740363 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.740649 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.740748 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.741879 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.741880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.741881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.741887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.743426 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.743840 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.743841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.743842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.743893 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.744104 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.744104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.744105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.744205 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.744290 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.744464 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.744543 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:38:44.744655 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.738173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.98.3:57081)
0924 17:38:44.738316 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.738329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.738394 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.738573 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.738573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.738735 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.738786 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.738787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.739659 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.739660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.740261 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.740264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.740751 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.740752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.740828 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.741361 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.741366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.741801 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.743516 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.743517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.743517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.743761 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.743969 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.743969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.743970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.744045 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.744290 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.744291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.744341 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.744425 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.744546 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.765666 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.758442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.98.62:35603)
0924 17:38:44.758504 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.758506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.758514 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:44.758541 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.758541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.758551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.758611 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.759234 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.759738 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.760570 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.760763 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.760829 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.763234 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.763236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.763237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.763245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.764709 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.765039 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.765040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.765041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.765093 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.765242 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.765243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.765243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.765342 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.765413 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.765505 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.765564 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:38:44.767683 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:44.769289 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:44.776492 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.771866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42857 (local address 127.0.0.1:44476)
0924 17:38:44.772234 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.772406 (+ 172us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.772473 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.772753 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.772756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.772758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:44.772849 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.772851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.774215 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.774216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.775331 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.775333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.776168 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.776169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.776226 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.776243 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.776330 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.776396 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":81,"threads_started":1}
I20260924 17:38:44.776510 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.772007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.0.98.2:42857)
0924 17:38:44.772259 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.772264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.772471 (+ 207us) server_negotiation.cc:407] Connection header received
0924 17:38:44.772594 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.772598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.772616 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.772925 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:44.772982 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.774115 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.775408 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.776093 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.776154 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.776240 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.776374 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.776413 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260924 17:38:44.776647 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.772404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.98.1:45517)
0924 17:38:44.772618 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.772623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.772634 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:38:44.772741 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.772743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.772757 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.772834 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:44.773020 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.773727 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.775249 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.776024 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.776107 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.776197 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.776444 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.776486 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260924 17:38:44.777041 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.772266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45517 (local address 127.0.0.1:51814)
0924 17:38:44.772574 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.772585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.772645 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.772899 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.772900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.772902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:44.772937 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.772939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.773841 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.773842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.775146 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.775149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.776107 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.776108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.776226 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.776243 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.776360 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.776398 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260924 17:38:44.777343 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a (DEFAULT_TABLE table=test-table [id=f99d0d9e917046b6a22b0c6be0e96490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.777540 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.772102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45053 (local address 127.0.0.1:40128)
0924 17:38:44.772385 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.772405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.772483 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.772924 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.772925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.772927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:44.773004 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.773006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.774242 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.774243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.775148 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.775151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.777308 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.777309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.777374 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.777378 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.777435 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.777475 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":71,"threads_started":1}
I20260924 17:38:44.777792 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee08c605ab64f1dbfcb2ffd075e759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.777747 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.772263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.0.98.3:45053)
0924 17:38:44.772713 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.772717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.772726 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:44.772763 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.772766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.772783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.772857 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:44.773072 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.774144 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.776239 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.777129 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.777206 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.777274 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.777469 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.777511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":98,"threads_started":1}
I20260924 17:38:44.777750 1636 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a (DEFAULT_TABLE table=test-table [id=f99d0d9e917046b6a22b0c6be0e96490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.778095 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee08c605ab64f1dbfcb2ffd075e759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.778373 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a (DEFAULT_TABLE table=test-table [id=f99d0d9e917046b6a22b0c6be0e96490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.778650 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee08c605ab64f1dbfcb2ffd075e759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.781375 2014 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Bootstrap starting.
I20260924 17:38:44.781945 2015 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Bootstrap starting.
I20260924 17:38:44.782141 2014 tablet_bootstrap.cc:654] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.782481 2014 log.cc:824] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:44.782580 2015 tablet_bootstrap.cc:654] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.782706 2016 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Bootstrap starting.
I20260924 17:38:44.782928 2015 log.cc:824] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:44.783353 2014 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: No bootstrap required, opened a new log
I20260924 17:38:44.783380 2016 tablet_bootstrap.cc:654] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.783452 2014 ts_tablet_manager.cc:1397] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.783677 2016 log.cc:824] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:44.783713 2015 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: No bootstrap required, opened a new log
I20260924 17:38:44.783769 2015 ts_tablet_manager.cc:1397] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.784387 2016 tablet_bootstrap.cc:492] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: No bootstrap required, opened a new log
I20260924 17:38:44.784471 2016 ts_tablet_manager.cc:1397] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.785989 2015 raft_consensus.cc:348] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.786158 2015 raft_consensus.cc:374] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.786198 2015 raft_consensus.cc:729] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5193e20be1764757bced8f6c54874ec4, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.786348 2015 consensus_queue.cc:260] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.786414 2016 raft_consensus.cc:348] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.786590 2016 raft_consensus.cc:374] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.786612 2015 ts_tablet_manager.cc:1428] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:38:44.786620 2016 raft_consensus.cc:729] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd5b072fd3f433b8ff3bbe12a5f8e74, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.786751 1851 heartbeater.cc:503] Master 127.0.98.62:35603 was elected leader, sending a full tablet report...
I20260924 17:38:44.786741 2016 consensus_queue.cc:260] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.787017 2016 ts_tablet_manager.cc:1428] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:38:44.786620 2014 raft_consensus.cc:348] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.787096 1991 heartbeater.cc:503] Master 127.0.98.62:35603 was elected leader, sending a full tablet report...
I20260924 17:38:44.787106 2014 raft_consensus.cc:374] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.787153 2014 raft_consensus.cc:729] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150c488c37e4db79088d8be184f3637, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.787264 2014 consensus_queue.cc:260] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.787629 1711 heartbeater.cc:503] Master 127.0.98.62:35603 was elected leader, sending a full tablet report...
I20260924 17:38:44.787693 2014 ts_tablet_manager.cc:1428] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260924 17:38:44.797077 1852 tablet.cc:2369] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:44.807662 2022 raft_consensus.cc:482] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:44.807787 2022 raft_consensus.cc:504] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.808080 2022 leader_election.cc:288] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:44.811369 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.808528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41179 (local address 127.0.98.2:42857)
0924 17:38:44.808577 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.808580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.808592 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:38:44.808630 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.808631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.808644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.808714 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:44.808896 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.809581 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.810592 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.810978 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.811077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.811090 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.811252 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.811297 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:38:44.811388 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.808340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42857 (local address 127.0.98.1:41179)
0924 17:38:44.808504 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.808519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.808579 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.808743 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.808744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.808745 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:44.808824 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.808826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.809662 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.809663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.810498 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.810500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.811083 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.811085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.811175 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.811180 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.811236 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.811271 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 17:38:44.811456 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.808604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57007 (local address 127.0.98.3:45053)
0924 17:38:44.808648 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.808650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.808739 (+ 89us) server_negotiation.cc:407] Connection header received
0924 17:38:44.808786 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.808787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.808800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.808879 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:44.809053 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.809746 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.810673 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.811092 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.811161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.811171 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.811339 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.811386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:38:44.811700 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.808472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45053 (local address 127.0.98.1:57007)
0924 17:38:44.808666 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.808675 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.808731 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.808897 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.808897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.808899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:44.808977 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.808979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.809827 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.809827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.810583 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.810585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.811169 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.811169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.811248 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.811252 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.811316 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.811357 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":112,"threads_started":1}
I20260924 17:38:44.811786 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee08c605ab64f1dbfcb2ffd075e759a" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4" is_pre_election: true
I20260924 17:38:44.811975 1805 raft_consensus.cc:2454] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:44.812206 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee08c605ab64f1dbfcb2ffd075e759a" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" is_pre_election: true
I20260924 17:38:44.812232 1598 leader_election.cc:302] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5193e20be1764757bced8f6c54874ec4, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:44.812367 1944 raft_consensus.cc:2454] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:44.812363 2022 raft_consensus.cc:2781] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:44.812412 2022 raft_consensus.cc:482] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:44.812466 2022 raft_consensus.cc:3037] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.813386 2022 raft_consensus.cc:504] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.813498 2022 leader_election.cc:288] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Requested vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:44.813668 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee08c605ab64f1dbfcb2ffd075e759a" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4"
I20260924 17:38:44.813694 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee08c605ab64f1dbfcb2ffd075e759a" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
I20260924 17:38:44.813753 1805 raft_consensus.cc:3037] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.813753 1944 raft_consensus.cc:3037] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.814613 1944 raft_consensus.cc:2454] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:44.814613 1805 raft_consensus.cc:2454] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:44.814769 1598 leader_election.cc:302] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fd5b072fd3f433b8ff3bbe12a5f8e74, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:44.814873 2022 raft_consensus.cc:2781] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:44.815070 2022 raft_consensus.cc:686] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 1 LEADER]: Becoming Leader. State: Replica: d150c488c37e4db79088d8be184f3637, State: Running, Role: LEADER
I20260924 17:38:44.815174 2022 consensus_queue.cc:237] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } }
I20260924 17:38:44.816015 1508 catalog_manager.cc:5495] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150c488c37e4db79088d8be184f3637 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } health_report { overall_health: HEALTHY } } }
I20260924 17:38:44.823915 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.821363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.98.1:45517)
0924 17:38:44.821426 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.821427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.821437 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:38:44.821532 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.821533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.821550 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.821608 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:44.821738 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.822214 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.823019 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.823246 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.823322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.823562 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:44.823665 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:44.823749 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.823805 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.823833 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:38:44.825814 1944 raft_consensus.cc:1261] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:44.825821 1805 raft_consensus.cc:1261] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:44.826205 2024 consensus_queue.cc:1035] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:44.826337 2022 consensus_queue.cc:1035] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:44.829241 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332952165683339264 to 7332952165643386880. Current Snapshot: MvccSnapshot[applied={T|T < 7332952165683339264}]
I20260924 17:38:44.830991 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332952165683339264 to 7332952165643386880. Current Snapshot: MvccSnapshot[applied={T|T < 7332952165683339264 or (T in {7332952165683339264})}]
I20260924 17:38:44.832876 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:44.832983 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:44.833823 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Sending DeleteTablet for 3 replicas of tablet 7ee08c605ab64f1dbfcb2ffd075e759a
I20260924 17:38:44.834205 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40128
I20260924 17:38:44.834251 1636 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51814
I20260924 17:38:44.834381 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee08c605ab64f1dbfcb2ffd075e759a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44476
I20260924 17:38:44.834506 2037 tablet_replica.cc:330] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: stopping tablet replica
I20260924 17:38:44.834605 2038 tablet_replica.cc:330] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: stopping tablet replica
I20260924 17:38:44.834702 2038 raft_consensus.cc:2229] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:44.834817 2038 raft_consensus.cc:2258] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:44.834863 2037 raft_consensus.cc:2229] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:44.834936 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:44.835114 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:44.835116 2037 raft_consensus.cc:2258] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:44.835283 2036 tablet_replica.cc:330] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: stopping tablet replica
I20260924 17:38:44.835367 2036 raft_consensus.cc:2229] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:44.835741 2036 raft_consensus.cc:2258] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:44.835970 2037 ts_tablet_manager.cc:1905] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:44.836135 2036 ts_tablet_manager.cc:1905] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:44.836596 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c1adadfee48ea95736acd21994112 (DEFAULT_TABLE table=test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.836705 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c1adadfee48ea95736acd21994112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.836709 1636 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c1adadfee48ea95736acd21994112 (DEFAULT_TABLE table=test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.836802 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c1adadfee48ea95736acd21994112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.838020 2015 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Bootstrap starting.
I20260924 17:38:44.838052 2014 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Bootstrap starting.
I20260924 17:38:44.838420 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c1adadfee48ea95736acd21994112 (DEFAULT_TABLE table=test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:44.838446 2015 tablet_bootstrap.cc:654] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.838480 2014 tablet_bootstrap.cc:654] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.838516 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c1adadfee48ea95736acd21994112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.840426 2016 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Bootstrap starting.
I20260924 17:38:44.840755 2014 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: No bootstrap required, opened a new log
I20260924 17:38:44.840816 2014 ts_tablet_manager.cc:1397] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:38:44.840870 2016 tablet_bootstrap.cc:654] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.840917 2015 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: No bootstrap required, opened a new log
I20260924 17:38:44.840962 2015 ts_tablet_manager.cc:1397] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:38:44.840981 2014 raft_consensus.cc:348] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.841061 2014 raft_consensus.cc:374] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.841084 2014 raft_consensus.cc:729] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150c488c37e4db79088d8be184f3637, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.841132 2014 consensus_queue.cc:260] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.841104 2015 raft_consensus.cc:348] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.841194 2015 raft_consensus.cc:374] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.841219 2015 raft_consensus.cc:729] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5193e20be1764757bced8f6c54874ec4, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.841224 2014 ts_tablet_manager.cc:1428] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.841262 2015 consensus_queue.cc:260] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.841369 2015 ts_tablet_manager.cc:1428] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:44.841533 2036 ts_tablet_manager.cc:1918] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:44.841609 2036 log.cc:1196] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/7ee08c605ab64f1dbfcb2ffd075e759a
I20260924 17:38:44.841961 2036 ts_tablet_manager.cc:1939] T 7ee08c605ab64f1dbfcb2ffd075e759a P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting consensus metadata
I20260924 17:38:44.840584 2037 ts_tablet_manager.cc:1918] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:44.842176 2037 log.cc:1196] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/7ee08c605ab64f1dbfcb2ffd075e759a
I20260924 17:38:44.842439 2037 ts_tablet_manager.cc:1939] T 7ee08c605ab64f1dbfcb2ffd075e759a P d150c488c37e4db79088d8be184f3637: Deleting consensus metadata
I20260924 17:38:44.842465 1494 catalog_manager.cc:4841] TS 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053): tablet 7ee08c605ab64f1dbfcb2ffd075e759a (table test-table [id=f99d0d9e917046b6a22b0c6be0e96490]) successfully deleted
I20260924 17:38:44.842777 2016 tablet_bootstrap.cc:492] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: No bootstrap required, opened a new log
I20260924 17:38:44.842777 2038 ts_tablet_manager.cc:1905] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:44.842834 2016 ts_tablet_manager.cc:1397] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.842864 1494 catalog_manager.cc:4841] TS d150c488c37e4db79088d8be184f3637 (127.0.98.1:45517): tablet 7ee08c605ab64f1dbfcb2ffd075e759a (table test-table [id=f99d0d9e917046b6a22b0c6be0e96490]) successfully deleted
I20260924 17:38:44.842968 2016 raft_consensus.cc:348] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.843067 2016 raft_consensus.cc:374] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.843091 2016 raft_consensus.cc:729] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd5b072fd3f433b8ff3bbe12a5f8e74, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.843134 2016 consensus_queue.cc:260] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.843214 2016 ts_tablet_manager.cc:1428] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.844506 2038 ts_tablet_manager.cc:1918] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:44.844568 2038 log.cc:1196] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/7ee08c605ab64f1dbfcb2ffd075e759a
I20260924 17:38:44.844741 2038 ts_tablet_manager.cc:1939] T 7ee08c605ab64f1dbfcb2ffd075e759a P 5193e20be1764757bced8f6c54874ec4: Deleting consensus metadata
I20260924 17:38:44.845261 1494 catalog_manager.cc:4841] TS 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857): tablet 7ee08c605ab64f1dbfcb2ffd075e759a (table test-table [id=f99d0d9e917046b6a22b0c6be0e96490]) successfully deleted
W20260924 17:38:44.887277 1712 tablet.cc:2369] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:44.916886 2024 raft_consensus.cc:482] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:44.916991 2024 raft_consensus.cc:504] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.917179 2024 leader_election.cc:288] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:44.917526 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c1adadfee48ea95736acd21994112" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4" is_pre_election: true
I20260924 17:38:44.917639 1805 raft_consensus.cc:2454] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:44.917615 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c1adadfee48ea95736acd21994112" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" is_pre_election: true
I20260924 17:38:44.917773 1944 raft_consensus.cc:2454] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:44.917788 1598 leader_election.cc:302] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5193e20be1764757bced8f6c54874ec4, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:44.917896 2024 raft_consensus.cc:2781] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:44.917935 2024 raft_consensus.cc:482] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:44.917959 2024 raft_consensus.cc:3037] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.918632 2024 raft_consensus.cc:504] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.918767 2024 leader_election.cc:288] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Requested vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:44.918941 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c1adadfee48ea95736acd21994112" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
I20260924 17:38:44.918936 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c1adadfee48ea95736acd21994112" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4"
I20260924 17:38:44.919013 1805 raft_consensus.cc:3037] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.919028 1944 raft_consensus.cc:3037] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:44.919698 1944 raft_consensus.cc:2454] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:44.919804 1805 raft_consensus.cc:2454] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:44.919857 1598 leader_election.cc:302] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fd5b072fd3f433b8ff3bbe12a5f8e74, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:44.919997 2024 raft_consensus.cc:2781] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:44.920068 2024 raft_consensus.cc:686] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 1 LEADER]: Becoming Leader. State: Replica: d150c488c37e4db79088d8be184f3637, State: Running, Role: LEADER
I20260924 17:38:44.920127 2024 consensus_queue.cc:237] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.920789 1507 catalog_manager.cc:5495] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150c488c37e4db79088d8be184f3637 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:38:44 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:44.966823 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.959822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35603 (local address 127.0.0.1:42732)
0924 17:38:44.959995 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:44.960006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:44.960076 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:44.960235 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:44.960236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:44.960541 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:44.960585 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.960587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.961348 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.961349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.961910 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:44.961912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.962267 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.962268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:44.962307 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.962838 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:44.962843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:44.964318 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:44.965784 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.965791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.965793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.965999 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.966187 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:44.966188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:44.966188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:44.966242 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:44.966501 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:44.966507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:44.966553 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 17:38:44.966619 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:38:44.966719 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260924 17:38:44.966892 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:44.959863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.0.98.62:35603)
0924 17:38:44.959931 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:44.959935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:44.960074 (+ 139us) server_negotiation.cc:407] Connection header received
0924 17:38:44.960140 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:44.960141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:44.960154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:44.960229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:44.960675 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.961221 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.962005 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:44.962187 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:44.962262 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:44.964410 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:44.964411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:44.964412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:44.964419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:44.965620 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.966071 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.966072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.966073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.966124 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:44.966304 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:44.966304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:44.966304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:44.966421 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:44.966472 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:44.966649 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 17:38:44.966763 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":42}
I20260924 17:38:44.969046 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:38:44.969229 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:44.970554 1636 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebc5afde97e43a889fd2b2808f92f44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a58cd8093064683bf4778c3b7f4794f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:44.970641 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebc5afde97e43a889fd2b2808f92f44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a58cd8093064683bf4778c3b7f4794f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:44.970626 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebc5afde97e43a889fd2b2808f92f44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a58cd8093064683bf4778c3b7f4794f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:44.970733 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebc5afde97e43a889fd2b2808f92f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.970764 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebc5afde97e43a889fd2b2808f92f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.970793 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebc5afde97e43a889fd2b2808f92f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:44.972085 2015 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: Bootstrap starting.
I20260924 17:38:44.972275 2014 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: Bootstrap starting.
I20260924 17:38:44.972536 2015 tablet_bootstrap.cc:654] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.972568 2014 tablet_bootstrap.cc:654] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.972878 2016 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Bootstrap starting.
I20260924 17:38:44.973220 2014 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: No bootstrap required, opened a new log
I20260924 17:38:44.973284 2014 ts_tablet_manager.cc:1397] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:44.973295 2016 tablet_bootstrap.cc:654] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:44.973412 2014 raft_consensus.cc:348] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.973889 2014 raft_consensus.cc:374] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.973917 2014 raft_consensus.cc:729] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150c488c37e4db79088d8be184f3637, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.973969 2014 consensus_queue.cc:260] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.974083 2014 ts_tablet_manager.cc:1428] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 17:38:44.974100 2024 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:44.974174 2024 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:44.974298 2015 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: No bootstrap required, opened a new log
I20260924 17:38:44.974360 2015 ts_tablet_manager.cc:1397] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.974527 2015 raft_consensus.cc:348] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.974606 2015 raft_consensus.cc:374] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.974606 2016 tablet_bootstrap.cc:492] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: No bootstrap required, opened a new log
I20260924 17:38:44.974628 2015 raft_consensus.cc:729] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5193e20be1764757bced8f6c54874ec4, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.974644 2016 ts_tablet_manager.cc:1397] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:44.974670 2015 consensus_queue.cc:260] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.974771 2015 ts_tablet_manager.cc:1428] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:44.974774 2016 raft_consensus.cc:348] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.974876 2016 raft_consensus.cc:374] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:44.974906 2016 raft_consensus.cc:729] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd5b072fd3f433b8ff3bbe12a5f8e74, State: Initialized, Role: FOLLOWER
I20260924 17:38:44.974911 2020 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:44.974959 2016 consensus_queue.cc:260] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:44.975008 2020 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:44.975080 2016 ts_tablet_manager.cc:1428] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:44.975193 2021 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:44.975286 2021 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 17:38:44.984432 1992 tablet.cc:2369] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:45.063441 2024 raft_consensus.cc:482] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:45.063572 2024 raft_consensus.cc:504] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:45.063768 2024 leader_election.cc:288] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:45.064028 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebc5afde97e43a889fd2b2808f92f44" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4" is_pre_election: true
I20260924 17:38:45.064076 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebc5afde97e43a889fd2b2808f92f44" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" is_pre_election: true
I20260924 17:38:45.064138 1805 raft_consensus.cc:2454] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:45.064148 1944 raft_consensus.cc:2454] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 0.
I20260924 17:38:45.064291 1598 leader_election.cc:302] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5193e20be1764757bced8f6c54874ec4, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:45.064410 2024 raft_consensus.cc:2781] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:45.064445 2024 raft_consensus.cc:482] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:45.064474 2024 raft_consensus.cc:3037] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:45.065127 2024 raft_consensus.cc:504] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:45.065264 2024 leader_election.cc:288] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Requested vote from peers 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857), 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053)
I20260924 17:38:45.065452 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebc5afde97e43a889fd2b2808f92f44" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5193e20be1764757bced8f6c54874ec4"
I20260924 17:38:45.065465 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebc5afde97e43a889fd2b2808f92f44" candidate_uuid: "d150c488c37e4db79088d8be184f3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74"
I20260924 17:38:45.065557 1805 raft_consensus.cc:3037] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:45.065554 1944 raft_consensus.cc:3037] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:45.066149 1944 raft_consensus.cc:2454] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:45.066167 1805 raft_consensus.cc:2454] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150c488c37e4db79088d8be184f3637 in term 1.
I20260924 17:38:45.066331 1598 leader_election.cc:302] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fd5b072fd3f433b8ff3bbe12a5f8e74, d150c488c37e4db79088d8be184f3637; no voters:
I20260924 17:38:45.066481 2024 raft_consensus.cc:2781] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:45.066538 2024 raft_consensus.cc:686] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [term 1 LEADER]: Becoming Leader. State: Replica: d150c488c37e4db79088d8be184f3637, State: Running, Role: LEADER
I20260924 17:38:45.066598 2024 consensus_queue.cc:237] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260924 17:38:45.066887 2022 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: TxnStatusTablet state changed. Reason: New leader d150c488c37e4db79088d8be184f3637. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.066998 2022 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:45.067181 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:38:45.067305 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:38:45.067263 1508 catalog_manager.cc:5495] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150c488c37e4db79088d8be184f3637 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } health_report { overall_health: UNKNOWN } } }
I20260924 17:38:45.092677 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.089972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45517 (local address 127.0.0.1:51828)
0924 17:38:45.090073 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:45.090087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:45.090174 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:45.090364 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:45.090365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:45.090366 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:45.090421 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:45.090424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.091196 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.091197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:45.091799 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:45.091801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.092179 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.092180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:45.092244 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.092296 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:45.092501 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:45.092510 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 17:38:45.092552 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 17:38:45.092605 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 17:38:45.092677 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.090091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.98.1:45517)
0924 17:38:45.090157 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:45.090160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:45.090169 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:45.090256 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:45.090257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:45.090270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:45.090358 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:45.090505 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.091098 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.091896 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.092096 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.092178 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.092388 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:45.092428 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:45.092498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:45.092569 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:38:45.092593 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:38:45.093945 1944 raft_consensus.cc:1261] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:45.094136 1805 raft_consensus.cc:1261] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:45.094223 2022 consensus_queue.cc:1035] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:45.094357 2024 consensus_queue.cc:1035] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:45.094830 2021 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: TxnStatusTablet state changed. Reason: New leader d150c488c37e4db79088d8be184f3637. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.094949 2021 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:45.095152 2020 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: TxnStatusTablet state changed. Reason: New leader d150c488c37e4db79088d8be184f3637. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.095260 2020 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:45.095623 2022 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.095700 2022 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:45.095748 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332952166782758912 to 7332952166673141760. Current Snapshot: MvccSnapshot[applied={T|T < 7332952166782758912}]
I20260924 17:38:45.096073 2024 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.096138 2024 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P d150c488c37e4db79088d8be184f3637: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:45.096241 2020 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.096333 2020 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 5193e20be1764757bced8f6c54874ec4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:45.096489 2021 tablet_replica.cc:439] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d150c488c37e4db79088d8be184f3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d150c488c37e4db79088d8be184f3637" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45517 } } peers { permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 } } peers { permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } } }
I20260924 17:38:45.096573 2021 tablet_replica.cc:442] T 8ebc5afde97e43a889fd2b2808f92f44 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:45.103170 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.100246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37935 (local address 127.0.98.1:45517)
0924 17:38:45.100285 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:45.100287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:45.100418 (+ 131us) server_negotiation.cc:407] Connection header received
0924 17:38:45.100451 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:45.100451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:45.100461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:45.100511 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:45.100637 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.101090 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.102189 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.102460 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.102528 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.102723 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:45.102772 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:45.102835 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:45.102999 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 17:38:45.103018 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:38:45.103276 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.100149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45517 (local address 127.0.98.1:37935)
0924 17:38:45.100324 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:45.100335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:45.100392 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:45.100531 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:45.100532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:45.100534 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:45.100584 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:45.100585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.101164 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.101165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:45.102092 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:45.102094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.102550 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.102551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:45.102607 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.102624 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:45.102921 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:45.102923 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:38:45.102990 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:38:45.103069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260924 17:38:45.106148 1944 raft_consensus.cc:1261] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:45.106321 1805 raft_consensus.cc:1261] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Refusing update from remote peer d150c488c37e4db79088d8be184f3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:45.106381 2058 consensus_queue.cc:1035] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd5b072fd3f433b8ff3bbe12a5f8e74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:45.106541 2022 consensus_queue.cc:1035] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5193e20be1764757bced8f6c54874ec4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:45.117631 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:45.117741 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:45.118498 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 341a27f0d59c478798850d31a9418616: Sending DeleteTablet for 3 replicas of tablet 4b9c1adadfee48ea95736acd21994112
I20260924 17:38:45.118759 1636 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9c1adadfee48ea95736acd21994112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51814
I20260924 17:38:45.118852 2037 tablet_replica.cc:330] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: stopping tablet replica
I20260924 17:38:45.118841 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9c1adadfee48ea95736acd21994112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40128
I20260924 17:38:45.118896 2037 raft_consensus.cc:2229] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:45.118841 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9c1adadfee48ea95736acd21994112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44476
I20260924 17:38:45.118968 2037 raft_consensus.cc:2258] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:45.118979 2038 tablet_replica.cc:330] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: stopping tablet replica
I20260924 17:38:45.119063 2038 raft_consensus.cc:2229] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:45.119098 2038 pending_rounds.cc:62] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Trying to abort 2 pending ops.
I20260924 17:38:45.119089 2036 tablet_replica.cc:330] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: stopping tablet replica
I20260924 17:38:45.119138 2036 raft_consensus.cc:2229] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:45.119119 2038 pending_rounds.cc:68] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332952166833106944 op_type: PARTICIPANT_OP participant_request { tablet_id: "4b9c1adadfee48ea95736acd21994112" op { type: BEGIN_TXN txn_id: 0 } }
I20260924 17:38:45.119207 2036 raft_consensus.cc:2258] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:45.119254 2038 pending_rounds.cc:68] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332952166073257984 op_type: NO_OP noop_request { }
I20260924 17:38:45.119295 2038 raft_consensus.cc:2866] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 17:38:45.119328 2038 raft_consensus.cc:2258] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 17:38:45.119688 2062 consensus_queue.cc:1159] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5193e20be1764757bced8f6c54874ec4" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260924 17:38:45.119720 2037 ts_tablet_manager.cc:1905] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:45.119920 2038 ts_tablet_manager.cc:1905] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:45.120465 2036 ts_tablet_manager.cc:1905] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:45.121480 2038 ts_tablet_manager.cc:1918] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:45.121538 2038 log.cc:1196] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/4b9c1adadfee48ea95736acd21994112
I20260924 17:38:45.121758 2038 ts_tablet_manager.cc:1939] T 4b9c1adadfee48ea95736acd21994112 P 5193e20be1764757bced8f6c54874ec4: Deleting consensus metadata
I20260924 17:38:45.122040 2036 ts_tablet_manager.cc:1918] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:45.122100 2036 log.cc:1196] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/4b9c1adadfee48ea95736acd21994112
I20260924 17:38:45.122311 2036 ts_tablet_manager.cc:1939] T 4b9c1adadfee48ea95736acd21994112 P 6fd5b072fd3f433b8ff3bbe12a5f8e74: Deleting consensus metadata
I20260924 17:38:45.122700 2037 ts_tablet_manager.cc:1918] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:45.122761 2037 log.cc:1196] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/4b9c1adadfee48ea95736acd21994112
I20260924 17:38:45.122787 1494 catalog_manager.cc:4841] TS 5193e20be1764757bced8f6c54874ec4 (127.0.98.2:42857): tablet 4b9c1adadfee48ea95736acd21994112 (table test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]) successfully deleted
I20260924 17:38:45.122965 2037 ts_tablet_manager.cc:1939] T 4b9c1adadfee48ea95736acd21994112 P d150c488c37e4db79088d8be184f3637: Deleting consensus metadata
I20260924 17:38:45.123086 1494 catalog_manager.cc:4841] TS 6fd5b072fd3f433b8ff3bbe12a5f8e74 (127.0.98.3:45053): tablet 4b9c1adadfee48ea95736acd21994112 (table test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]) successfully deleted
I20260924 17:38:45.123446 1494 catalog_manager.cc:4841] TS d150c488c37e4db79088d8be184f3637 (127.0.98.1:45517): tablet 4b9c1adadfee48ea95736acd21994112 (table test-table [id=b5a9a705e09142a0a5513fa3d29c1c62]) successfully deleted
Sep 24 17:38:45 dist-test-slave-d0pf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:45.132105 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.123410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.0.98.1:45517)
0924 17:38:45.123464 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:45.123466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:45.123652 (+ 186us) server_negotiation.cc:407] Connection header received
0924 17:38:45.123726 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:45.123727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:45.123737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:45.123793 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:45.124348 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.124894 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.125677 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.125898 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.125975 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.129128 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:45.129140 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:45.129141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:45.129148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:45.130817 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:45.131253 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:45.131257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:45.131257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:45.131323 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:45.131532 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:45.131533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:45.131533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:45.131688 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:45.131859 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:45.131902 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 17:38:45.131994 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 17:38:45.132507 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51842
I20260924 17:38:45.139907 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:45.133563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.98.62:35603)
0924 17:38:45.133613 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:45.133615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:45.133669 (+ 54us) server_negotiation.cc:407] Connection header received
0924 17:38:45.133745 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:45.133746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:45.133756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:45.133805 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:45.134307 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.134809 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.135706 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:45.135923 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:45.136001 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:45.137258 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:45.137260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:45.137260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:45.137268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:45.138751 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:45.139126 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:45.139127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:45.139128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:45.139190 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:45.139396 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:45.139396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:45.139397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:45.139516 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:45.139627 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:45.139722 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:38:45.139802 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:38:45.140249 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42746
I20260924 17:38:45.140957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1580
I20260924 17:38:45.147787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1720
I20260924 17:38:45.152598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1860
I20260924 17:38:45.157279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 1476
2026-09-24T17:38:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2321 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:45 dist-test-slave-d0pf krb5kdc[2071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:45 dist-test-slave-d0pf krb5kdc[2071](info): set up 2 sockets
Sep 24 17:38:45 dist-test-slave-d0pf krb5kdc[2071](info): commencing operation
krb5kdc: starting...
W20260924 17:38:46.206943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:46.325884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:46.406891 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:46.407173 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:46.407202 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:46.408304 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:46.408340 2084 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:38:46.408353 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:46.408366 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:46.408378 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:46.408390 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:46.409785 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39635
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:46.410032 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:46.410284 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:46.414997 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.415017 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.415112 2084 server_base.cc:1034] running on GCE node
W20260924 17:38:46.415010 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.415539 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:46.415870 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:46.417033 2084 hybrid_clock.cc:648] HybridClock initialized: now 1790271526417027 us; error 40 us; skew 500 ppm
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:46.419512 2084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:46.419993 2084 webserver.cc:460] Webserver started at http://127.0.98.62:45165/ using document root <none> and password file <none>
I20260924 17:38:46.420228 2084 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:46.420298 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:46.420460 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:46.421489 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "2a9d05c6073b4beabf8b6f9fb70445b5"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "a2b72e1a004e4cee718bd19abb3bf17a"
server_key_iv: "f5598bfa9a5727cced0f76d12e16cae8"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.421885 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a9d05c6073b4beabf8b6f9fb70445b5"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "a2b72e1a004e4cee718bd19abb3bf17a"
server_key_iv: "f5598bfa9a5727cced0f76d12e16cae8"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.423250 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:38:46.423935 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.424109 2084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:46.424211 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "2a9d05c6073b4beabf8b6f9fb70445b5"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "a2b72e1a004e4cee718bd19abb3bf17a"
server_key_iv: "f5598bfa9a5727cced0f76d12e16cae8"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.424291 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:46.432444 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:46.434681 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:46.434841 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:46.439522 2084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39635
I20260924 17:38:46.439560 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39635 every 8 connection(s)
I20260924 17:38:46.439885 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:46.440424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2084
I20260924 17:38:46.440521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:46.440567 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:46.440721 392 external_mini_cluster.cc:1428] Setting key 889d04302a6466c45ba1fbb09111db50
I20260924 17:38:46.443367 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Bootstrap starting.
I20260924 17:38:46.443958 2162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:46.444237 2162 log.cc:824] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:46.444890 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: No bootstrap required, opened a new log
I20260924 17:38:46.446718 2162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260924 17:38:46.446903 2162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:46.446939 2162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9d05c6073b4beabf8b6f9fb70445b5, State: Initialized, Role: FOLLOWER
I20260924 17:38:46.447109 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260924 17:38:46.447193 2162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:46.447222 2162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:46.447269 2162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:46.447794 2162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260924 17:38:46.447875 2162 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9d05c6073b4beabf8b6f9fb70445b5; no voters:
I20260924 17:38:46.448083 2162 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:46.448113 2167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:46.448228 2167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [term 1 LEADER]: Becoming Leader. State: Replica: 2a9d05c6073b4beabf8b6f9fb70445b5, State: Running, Role: LEADER
I20260924 17:38:46.448342 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:46.448369 2167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260924 17:38:46.448750 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260924 17:38:46.448899 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9d05c6073b4beabf8b6f9fb70445b5. Latest consensus state: current_term: 1 leader_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9d05c6073b4beabf8b6f9fb70445b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260924 17:38:46.449795 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:46.449944 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5 [sys.catalog]: This master's current role is: LEADER
W20260924 17:38:46.450241 2182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:38:46.450314 2182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:38:46.450392 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:46.450555 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:46.451669 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.441199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.0.98.62:39635)
0924 17:38:46.441388 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.441397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.441431 (+ 34us) server_negotiation.cc:407] Connection header received
0924 17:38:46.441924 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.441931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.441979 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.442133 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.442743 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.443789 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.444692 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.445141 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.445270 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.447804 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.447808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.447808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.447820 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.449706 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.450143 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.450145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.450146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.450246 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.450444 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.450445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.450445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.450743 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.450867 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.451335 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.451473 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260924 17:38:46.452104 2184 catalog_manager.cc:1347] Generated new cluster ID: c91c659f996c4479a98dd19639baf8c3
I20260924 17:38:46.452152 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:38:46.472155 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:46.472931 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:46.477526 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Generated new TSK 0
I20260924 17:38:46.477720 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:46.505996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:46.587469 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:46.587693 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:46.587719 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:46.589017 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:46.589058 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:46.589103 2188 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:38:46.589130 2188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:46.589157 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:46.590721 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:46.590993 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:46.591259 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:46.596381 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.596388 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.596398 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.596529 2188 server_base.cc:1034] running on GCE node
I20260924 17:38:46.596832 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:46.597090 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:46.598242 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790271526598229 us; error 28 us; skew 500 ppm
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:46.600790 2188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:46.601269 2188 webserver.cc:460] Webserver started at http://127.0.98.1:39121/ using document root <none> and password file <none>
I20260924 17:38:46.601508 2188 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:46.601554 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:46.601723 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:46.602785 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b93563c341c4d00bbc600ccf6c91357"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "813fa5704a55460a77e2659d46bdc120"
server_key_iv: "aaf18f0701e5477d6b13a7dba32f86a9"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.603185 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b93563c341c4d00bbc600ccf6c91357"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "813fa5704a55460a77e2659d46bdc120"
server_key_iv: "aaf18f0701e5477d6b13a7dba32f86a9"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.604487 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:46.605226 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.605417 2188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:46.605508 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "5b93563c341c4d00bbc600ccf6c91357"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "813fa5704a55460a77e2659d46bdc120"
server_key_iv: "aaf18f0701e5477d6b13a7dba32f86a9"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.605581 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:46.616652 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:46.618948 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:46.619138 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:46.619633 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:46.619706 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.619755 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:46.619781 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.626719 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43375
I20260924 17:38:46.626802 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43375 every 8 connection(s)
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:46.627208 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:46.630388 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.620181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48437 (local address 127.0.98.62:39635)
0924 17:38:46.620256 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.620259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.620492 (+ 233us) server_negotiation.cc:407] Connection header received
0924 17:38:46.621015 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.621017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.621034 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.621119 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.622061 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.622907 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.624590 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.624941 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.625034 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.627749 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.627750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.627751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.627759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.629095 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.629523 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.629524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.629525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.629597 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.629852 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.629853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.629853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.629963 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.630014 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.630254 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.630325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:38:46.630594 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.620174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:48437)
0924 17:38:46.620352 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.620392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.620911 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.621222 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.621226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.621638 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.621942 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.621949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.623029 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.623031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.624481 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.624485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.625058 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.625059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.625174 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.625789 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.625799 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.627643 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.629190 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.629196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.629198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.629417 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.629677 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.629677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.629678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.629787 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.630021 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.630026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.630144 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.630227 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.630349 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":116,"threads_started":1}
I20260924 17:38:46.630900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2188
I20260924 17:38:46.631024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:46.631116 2319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260924 17:38:46.631202 2319 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:46.631250 392 external_mini_cluster.cc:1428] Setting key ab158f5a607f6c205dc84fb76c97eb0a
I20260924 17:38:46.631399 2319 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260924 17:38:46.631919 2116 ts_manager.cc:194] Registered new tserver with Master: 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375)
I20260924 17:38:46.632896 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48437
I20260924 17:38:46.638850 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.632257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38187 (local address 127.0.98.62:39635)
0924 17:38:46.632304 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.632306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.632469 (+ 163us) server_negotiation.cc:407] Connection header received
0924 17:38:46.632501 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.632502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.632513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.632586 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.632911 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.633741 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.634476 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.634685 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.634775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.636170 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.636171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.636172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.636178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.637656 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.638097 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.638098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.638099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.638156 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.638379 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.638379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.638380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.638483 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.638534 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.638715 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.638787 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:38:46.638940 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:38187)
0924 17:38:46.632393 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.632403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.632460 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.632591 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.632592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.632766 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.632837 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.632838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.633843 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.633845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.634368 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.634370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.634777 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.634778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.634859 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.635544 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.635550 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.636017 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.637746 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.637747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.637747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.638008 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.638236 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.638238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.638238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.638311 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.638552 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.638553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.638604 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.638699 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.638817 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:46.658810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:46.746289 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:46.746551 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:46.746592 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:46.747910 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:46.747974 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:46.748010 2328 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:38:46.748035 2328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:46.748072 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:46.749739 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:46.750018 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:46.750276 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:46.755294 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.755311 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.755300 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.755620 2328 server_base.cc:1034] running on GCE node
I20260924 17:38:46.755831 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:46.756111 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:46.757277 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790271526757269 us; error 40 us; skew 500 ppm
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:46.759753 2328 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:46.760233 2328 webserver.cc:460] Webserver started at http://127.0.98.2:44577/ using document root <none> and password file <none>
I20260924 17:38:46.760466 2328 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:46.760532 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:46.760694 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:46.761772 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "71881b8589eb426982ff85e43eca261b"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "88c7a01ed164da38d286763ec16a77c8"
server_key_iv: "aed893afdccb48ca26e0e6d07c3a4a92"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.762147 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71881b8589eb426982ff85e43eca261b"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "88c7a01ed164da38d286763ec16a77c8"
server_key_iv: "aed893afdccb48ca26e0e6d07c3a4a92"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.763620 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:38:46.764344 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.764509 2328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:46.764631 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "71881b8589eb426982ff85e43eca261b"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "88c7a01ed164da38d286763ec16a77c8"
server_key_iv: "aed893afdccb48ca26e0e6d07c3a4a92"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.764706 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:46.776506 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:46.778398 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:46.778571 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:46.779194 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:46.779263 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.779305 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:46.779328 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:46.786507 2328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40867
I20260924 17:38:46.786586 2458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40867 every 8 connection(s)
I20260924 17:38:46.786926 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:46.789228 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.779568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42893 (local address 127.0.98.62:39635)
0924 17:38:46.779621 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.779624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.779907 (+ 283us) server_negotiation.cc:407] Connection header received
0924 17:38:46.780420 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.780422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.780434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.780481 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.781195 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.781735 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.782756 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.783021 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.783129 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.786568 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.786570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.786570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.786578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.788024 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.788385 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.788385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.788386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.788433 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.788631 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.788631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.788632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.788729 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.788778 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.789106 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.789166 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:38:46.789435 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.779580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:42893)
0924 17:38:46.779774 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.779814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.780319 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.780555 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.780558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.780880 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.781080 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.781086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.781833 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.781834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.782662 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.782664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.783148 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.783149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.783278 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.783911 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.783922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.786454 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.788097 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.788103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.788104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.788319 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.788473 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.788473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.788473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.788581 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.788785 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.788790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.788957 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.789084 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.789238 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":128,"threads_started":1}
I20260924 17:38:46.789922 2459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260924 17:38:46.790000 2459 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:46.790196 2459 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260924 17:38:46.790598 2116 ts_manager.cc:194] Registered new tserver with Master: 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867)
I20260924 17:38:46.791189 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42893
I20260924 17:38:46.793890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2328
I20260924 17:38:46.793985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:46.794193 392 external_mini_cluster.cc:1428] Setting key a2ed8a34fb4ef012f8ac5c14eb405de2
I20260924 17:38:46.796700 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.790967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:55145)
0924 17:38:46.791122 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.791132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.791218 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.791392 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.791392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.791610 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.791675 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.791677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.792285 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.792285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.792798 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.792799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.793103 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.793104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.793167 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.793591 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.793596 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.794043 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.795610 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.795611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.795612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.795843 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.796027 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.796028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.796028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.796097 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.796345 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.796345 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.796388 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.796475 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.796597 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260924 17:38:46.796743 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.791016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55145 (local address 127.0.98.62:39635)
0924 17:38:46.791068 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.791070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.791203 (+ 133us) server_negotiation.cc:407] Connection header received
0924 17:38:46.791275 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.791275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.791286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.791337 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.791753 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.792197 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.792867 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.793039 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.793093 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.794131 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.794132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.794133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.794138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.795529 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.795916 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.795916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.795917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.795965 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.796155 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.796156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.796156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.796284 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.796335 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.796518 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.796626 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:46.821655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:46.903951 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:46.904166 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:46.904191 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:46.905349 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:46.905390 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:46.905411 2468 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:38:46.905424 2468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:38:46.905441 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:46.906877 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:46.907155 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:46.907409 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:46.912596 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:46.912607 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.912688 2468 server_base.cc:1034] running on GCE node
W20260924 17:38:46.912607 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:46.913019 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:46.913278 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:46.914424 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790271526914411 us; error 27 us; skew 500 ppm
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:46.916855 2468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:46.917317 2468 webserver.cc:460] Webserver started at http://127.0.98.3:43717/ using document root <none> and password file <none>
I20260924 17:38:46.917537 2468 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:46.917613 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:46.917762 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:46.918799 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "b3ff38f71473424aa85221696186efde"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "97d61b2a71470e8be3a8e510273ffcae"
server_key_iv: "b692d745f275e0076da6f234fea5577c"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.919198 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b3ff38f71473424aa85221696186efde"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "97d61b2a71470e8be3a8e510273ffcae"
server_key_iv: "b692d745f275e0076da6f234fea5577c"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.920506 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:46.921244 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.921422 2468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:46.921525 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "b3ff38f71473424aa85221696186efde"
format_stamp: "Formatted at 2026-09-24 17:38:46 on dist-test-slave-d0pf"
server_key: "97d61b2a71470e8be3a8e510273ffcae"
server_key_iv: "b692d745f275e0076da6f234fea5577c"
server_key_version: "encryptionkey@0"
I20260924 17:38:46.921598 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:46.929170 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:46.931200 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:46.931360 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:46.931919 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:46.931979 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:46.932022 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:46.932049 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:46.939294 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43477
I20260924 17:38:46.939741 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:38:46.939752 2598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43477 every 8 connection(s)
I20260924 17:38:46.941592 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.932352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36127 (local address 127.0.98.62:39635)
0924 17:38:46.932428 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.932430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.932694 (+ 264us) server_negotiation.cc:407] Connection header received
0924 17:38:46.933160 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.933162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.933175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.933262 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.933965 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.934456 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.935809 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.936169 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.936254 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.938569 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.938570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.938571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.938579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.940390 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.940803 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.940804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.940804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.940855 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.941066 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.941066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.941066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.941177 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.941228 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.941469 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.941530 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:38:46.941793 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.932363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:36127)
0924 17:38:46.932543 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.932599 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.933067 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.933368 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.933372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.933672 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.933856 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.933861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.934558 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.934559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.935726 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.935731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.936289 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.936290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.936441 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.937054 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.937063 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.938458 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.940458 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.940462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.940464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.940733 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.940913 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.940914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.940914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.941011 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.941232 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.941236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.941361 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.941443 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.941560 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260924 17:38:46.942229 2599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260924 17:38:46.942308 2599 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:46.942485 2599 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260924 17:38:46.942893 2115 ts_manager.cc:194] Registered new tserver with Master: b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:46.943418 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36127
I20260924 17:38:46.946444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2468
I20260924 17:38:46.946554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:46.946789 392 external_mini_cluster.cc:1428] Setting key bdfc31005b6d24a1c982cf3a0d15d684
I20260924 17:38:46.947731 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:38:46.948923 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.943370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:43747)
0924 17:38:46.943557 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.943567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.943640 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.943745 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.943745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.943918 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:46.943979 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.943980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.944644 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.944644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.945168 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.945169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.945459 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.945460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.945512 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.945867 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:46.945871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:46.946140 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:46.947965 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.947966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.947966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.948126 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.948339 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:46.948340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:46.948340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:46.948410 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:46.948692 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:46.948693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:46.948729 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.948781 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.948855 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260924 17:38:46.949033 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.943433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43747 (local address 127.0.98.62:39635)
0924 17:38:46.943471 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.943473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.943626 (+ 153us) server_negotiation.cc:407] Connection header received
0924 17:38:46.943694 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.943694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.943702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.943740 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.944050 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.944562 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.945229 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.945399 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.945449 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.946213 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.946214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.946214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.946219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.947894 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.948180 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.948181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.948181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.948244 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.948473 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.948474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.948474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.948621 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.948690 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.948821 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.948900 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:38:46 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:46.970644 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.963920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.0.98.62:39635)
0924 17:38:46.963979 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.963981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.964012 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:38:46.964034 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.964035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.964043 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.964084 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:46.964650 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.965135 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.965877 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.966048 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.966096 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.968350 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:46.968351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:46.968351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:46.968358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:46.969760 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.970050 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.970050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.970051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.970099 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:46.970258 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:46.970259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:46.970259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:46.970355 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:46.970427 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.970506 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.970558 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:38:46.972293 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:46.973815 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:46.980252 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.975945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.0.1:48494)
0924 17:38:46.976164 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.976205 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.976273 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.976581 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.976584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.976586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:46.976652 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.976654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.977837 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.977838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.979176 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.979178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.979918 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.979919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.979997 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.980010 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.980108 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.980163 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
I20260924 17:38:46.980262 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.976288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.0.98.2:40867)
0924 17:38:46.976430 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.976434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.976442 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:46.976477 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.976478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.976490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.976579 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:46.976731 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.977754 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.979262 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.979817 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.979916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.979980 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.980126 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.980163 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 17:38:46.980489 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.975960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43375 (local address 127.0.0.1:36588)
0924 17:38:46.976716 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.976727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.976774 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.977091 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.977092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.977093 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:46.977159 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.977160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.978323 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.978324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.979322 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.979324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.980240 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.980241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.980309 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.980313 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.980359 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.980410 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":148,"threads_started":1}
I20260924 17:38:46.980827 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.975947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43477 (local address 127.0.0.1:59498)
0924 17:38:46.976166 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:46.976205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:46.976273 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:46.976622 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:46.976623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:46.976625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:46.976683 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.976685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.977721 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.977722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.979177 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:46.979178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.980176 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.980177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:46.980255 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.980259 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:46.980724 (+ 465us) client_negotiation.cc:240] Negotiation successful
0924 17:38:46.980769 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":57,"thread_start_us":69,"threads_started":1}
I20260924 17:38:46.981070 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.976023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.98.3:43477)
0924 17:38:46.976450 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.976454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.976462 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:46.976493 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.976495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.976507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.976604 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:46.976773 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.977640 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.979278 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.980001 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.980072 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.980139 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.980931 (+ 792us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.980973 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":80,"threads_started":1}
I20260924 17:38:46.981050 2393 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab (DEFAULT_TABLE table=test-table [id=4713342818df46f38cbca308b4d49c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:46.981472 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c30ae2ef44f18be8fedff53e287ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:46.981751 2533 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab (DEFAULT_TABLE table=test-table [id=4713342818df46f38cbca308b4d49c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:46.982082 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c30ae2ef44f18be8fedff53e287ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:46.983309 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:46.976082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.98.1:43375)
0924 17:38:46.976254 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:46.976259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:46.976893 (+ 634us) server_negotiation.cc:407] Connection header received
0924 17:38:46.976946 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:46.976948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:46.976963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:46.977027 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:46.977252 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.978244 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.979406 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:46.980099 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:46.980162 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:46.980212 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:46.983189 (+ 2977us) server_negotiation.cc:299] Negotiation successful
0924 17:38:46.983234 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260924 17:38:46.983955 2253 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab (DEFAULT_TABLE table=test-table [id=4713342818df46f38cbca308b4d49c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:46.984282 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c30ae2ef44f18be8fedff53e287ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:46.985293 2619 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Bootstrap starting.
I20260924 17:38:46.985580 2620 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Bootstrap starting.
I20260924 17:38:46.985929 2619 tablet_bootstrap.cc:654] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:46.986284 2620 tablet_bootstrap.cc:654] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:46.986276 2619 log.cc:824] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:46.986553 2620 log.cc:824] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:46.987145 2619 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: No bootstrap required, opened a new log
I20260924 17:38:46.987291 2620 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: No bootstrap required, opened a new log
I20260924 17:38:46.987361 2620 ts_tablet_manager.cc:1397] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:38:46.987213 2619 ts_tablet_manager.cc:1397] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:46.988765 2620 raft_consensus.cc:348] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.988919 2620 raft_consensus.cc:374] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:46.988965 2620 raft_consensus.cc:729] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ff38f71473424aa85221696186efde, State: Initialized, Role: FOLLOWER
I20260924 17:38:46.988986 2619 raft_consensus.cc:348] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.989081 2620 consensus_queue.cc:260] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.989147 2619 raft_consensus.cc:374] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:46.989205 2619 raft_consensus.cc:729] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Initialized, Role: FOLLOWER
I20260924 17:38:46.989341 2620 ts_tablet_manager.cc:1428] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:38:46.989401 2599 heartbeater.cc:503] Master 127.0.98.62:39635 was elected leader, sending a full tablet report...
I20260924 17:38:46.989337 2619 consensus_queue.cc:260] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.989651 2619 ts_tablet_manager.cc:1428] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:46.989671 2459 heartbeater.cc:503] Master 127.0.98.62:39635 was elected leader, sending a full tablet report...
I20260924 17:38:46.989840 2623 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Bootstrap starting.
I20260924 17:38:46.990600 2623 tablet_bootstrap.cc:654] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:46.990893 2623 log.cc:824] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:46.991731 2623 tablet_bootstrap.cc:492] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: No bootstrap required, opened a new log
I20260924 17:38:46.991815 2623 ts_tablet_manager.cc:1397] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:38:46.993186 2623 raft_consensus.cc:348] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.993320 2623 raft_consensus.cc:374] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:46.993342 2623 raft_consensus.cc:729] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Initialized, Role: FOLLOWER
I20260924 17:38:46.993422 2623 consensus_queue.cc:260] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:46.993610 2623 ts_tablet_manager.cc:1428] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:38:46.993667 2319 heartbeater.cc:503] Master 127.0.98.62:39635 was elected leader, sending a full tablet report...
W20260924 17:38:47.037539 2460 tablet.cc:2369] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:47.100620 2624 raft_consensus.cc:482] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:47.100751 2624 raft_consensus.cc:504] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:47.101089 2624 leader_election.cc:288] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375)
I20260924 17:38:47.105051 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.101394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58807 (local address 127.0.98.1:43375)
0924 17:38:47.101436 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.101438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.101444 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:38:47.101477 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.101477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.101487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.101560 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.101782 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.102522 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104108 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.104684 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104761 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.104772 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.104938 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.104976 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:38:47.105048 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.101317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39191 (local address 127.0.98.2:40867)
0924 17:38:47.101366 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.101368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.101534 (+ 166us) server_negotiation.cc:407] Connection header received
0924 17:38:47.101582 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.101583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.101596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.101648 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.101798 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.102559 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104108 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.104688 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104761 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.104772 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.104938 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.104973 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:38:47.105069 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.101261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43375 (local address 127.0.98.3:58807)
0924 17:38:47.101300 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.101314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.101364 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.101589 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.101590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.101591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.101687 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.101689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.102649 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.102651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.103994 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.103997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.104775 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.104855 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.104862 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.104920 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.104971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":90}
I20260924 17:38:47.105077 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.101273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.98.3:39191)
0924 17:38:47.101438 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.101447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.101514 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.101673 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.101675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.101676 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.101728 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.101730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.102649 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.102651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.103994 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.103997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.104776 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.104777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.104855 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.104862 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.104920 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.104971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":62,"thread_start_us":109,"threads_started":1}
I20260924 17:38:47.105540 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c30ae2ef44f18be8fedff53e287ab" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357" is_pre_election: true
I20260924 17:38:47.105741 2273 raft_consensus.cc:2454] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 0.
I20260924 17:38:47.105784 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c30ae2ef44f18be8fedff53e287ab" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b" is_pre_election: true
I20260924 17:38:47.105928 2413 raft_consensus.cc:2454] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 0.
I20260924 17:38:47.105993 2484 leader_election.cc:302] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:47.106120 2624 raft_consensus.cc:2781] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:47.106199 2624 raft_consensus.cc:482] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:47.106216 2624 raft_consensus.cc:3037] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.106899 2624 raft_consensus.cc:504] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:47.107074 2624 leader_election.cc:288] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 election: Requested vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375)
I20260924 17:38:47.107231 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c30ae2ef44f18be8fedff53e287ab" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357"
I20260924 17:38:47.107228 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c30ae2ef44f18be8fedff53e287ab" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b"
I20260924 17:38:47.107312 2413 raft_consensus.cc:3037] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.107333 2273 raft_consensus.cc:3037] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.108080 2273 raft_consensus.cc:2454] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 1.
I20260924 17:38:47.108206 2413 raft_consensus.cc:2454] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 1.
I20260924 17:38:47.108297 2484 leader_election.cc:302] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:47.108392 2624 raft_consensus.cc:2781] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:47.108538 2624 raft_consensus.cc:686] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 1 LEADER]: Becoming Leader. State: Replica: b3ff38f71473424aa85221696186efde, State: Running, Role: LEADER
I20260924 17:38:47.108644 2624 consensus_queue.cc:237] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } }
I20260924 17:38:47.109647 2116 catalog_manager.cc:5495] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ff38f71473424aa85221696186efde (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b3ff38f71473424aa85221696186efde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } health_report { overall_health: UNKNOWN } } }
W20260924 17:38:47.127972 2320 tablet.cc:2369] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 17:38:47 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:47.146849 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.139848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.0.98.62:39635)
0924 17:38:47.139915 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.139917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.140066 (+ 149us) server_negotiation.cc:407] Connection header received
0924 17:38:47.140112 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.140113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.140122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.140171 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:47.140594 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.141086 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.141757 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.141924 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.142006 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.144178 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:47.144180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:47.144180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:47.144188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:47.145647 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:47.146060 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:47.146061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:47.146061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:47.146116 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:47.146318 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:47.146320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:47.146320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:47.146430 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:47.146498 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.146660 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.146741 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:38:47.146922 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.139809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.0.1:58098)
0924 17:38:47.139990 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.140001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.140068 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.140188 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.140189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.140486 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:47.140533 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.140535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.141154 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.141154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.141674 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.141676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.142004 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.142005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.142063 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.142625 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:47.142635 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:47.144081 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:47.145768 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:47.145775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:47.145777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:47.145967 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:47.146194 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:47.146195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:47.146195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:47.146257 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:47.146500 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:47.146507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:47.146554 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.146638 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.146797 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":56,"thread_start_us":113,"threads_started":1}
I20260924 17:38:47.149441 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:38:47.149669 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:47.151146 2253 tablet_service.cc:1467] Processing CreateTablet for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0aae7fcf55b4b66b7146edd748ee9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:47.151162 2393 tablet_service.cc:1467] Processing CreateTablet for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0aae7fcf55b4b66b7146edd748ee9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:47.151290 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.151347 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.151364 2533 tablet_service.cc:1467] Processing CreateTablet for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0aae7fcf55b4b66b7146edd748ee9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:38:47.151503 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37b87ffb8b04ef2bcc9e47fc464e5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.152488 2623 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: Bootstrap starting.
I20260924 17:38:47.152860 2623 tablet_bootstrap.cc:654] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.152977 2620 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: Bootstrap starting.
I20260924 17:38:47.153393 2620 tablet_bootstrap.cc:654] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.153523 2623 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: No bootstrap required, opened a new log
I20260924 17:38:47.153597 2623 ts_tablet_manager.cc:1397] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:47.153760 2619 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: Bootstrap starting.
I20260924 17:38:47.153769 2623 raft_consensus.cc:348] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.153861 2623 raft_consensus.cc:374] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.153898 2623 raft_consensus.cc:729] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.153959 2623 consensus_queue.cc:260] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.154084 2623 ts_tablet_manager.cc:1428] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:47.154161 2627 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.154232 2627 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.154417 2619 tablet_bootstrap.cc:654] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.154889 2620 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: No bootstrap required, opened a new log
I20260924 17:38:47.154944 2620 ts_tablet_manager.cc:1397] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:38:47.155186 2619 tablet_bootstrap.cc:492] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: No bootstrap required, opened a new log
I20260924 17:38:47.155139 2620 raft_consensus.cc:348] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.155224 2620 raft_consensus.cc:374] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.155236 2619 ts_tablet_manager.cc:1397] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:47.155251 2620 raft_consensus.cc:729] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ff38f71473424aa85221696186efde, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.155311 2620 consensus_queue.cc:260] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.155376 2619 raft_consensus.cc:348] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.155418 2620 ts_tablet_manager.cc:1428] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:47.155450 2619 raft_consensus.cc:374] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.155493 2619 raft_consensus.cc:729] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.155464 2624 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.155545 2619 consensus_queue.cc:260] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.155581 2624 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.155655 2619 ts_tablet_manager.cc:1428] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:47.155814 2625 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.155918 2625 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 17:38:47.190376 2600 tablet.cc:2369] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:47.257361 2625 raft_consensus.cc:482] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:47.257474 2625 raft_consensus.cc:504] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.257906 2625 leader_election.cc:288] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:47.260699 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.258104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43375 (local address 127.0.98.2:37203)
0924 17:38:47.258170 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.258183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.258248 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.258406 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.258407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.258409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.258482 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.258484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.259157 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.259158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.259910 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.259912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.260439 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.260520 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.260523 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.260585 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.260628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 17:38:47.260699 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.258234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37203 (local address 127.0.98.1:43375)
0924 17:38:47.258280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.258282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.258290 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:47.258320 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.258320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.258329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.258402 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.258553 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.259025 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260005 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.260359 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260442 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.260451 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.260602 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.260639 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:38:47.260881 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.258251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54841 (local address 127.0.98.3:43477)
0924 17:38:47.258299 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.258301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.258400 (+ 99us) server_negotiation.cc:407] Connection header received
0924 17:38:47.258480 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.258481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.258494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.258565 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.258729 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.259280 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260165 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.260536 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260618 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.260630 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.260781 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.260816 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:38:47.260906 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.258104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43477 (local address 127.0.98.2:54841)
0924 17:38:47.258316 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.258326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.258411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.258567 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.258568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.258570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.258643 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.258658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.259355 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.259356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.260073 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.260075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.260635 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.260636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.260710 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.260713 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.260778 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.260814 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":15,"thread_start_us":126,"threads_started":1}
I20260924 17:38:47.260964 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37b87ffb8b04ef2bcc9e47fc464e5a3" candidate_uuid: "71881b8589eb426982ff85e43eca261b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357" is_pre_election: true
I20260924 17:38:47.261065 2273 raft_consensus.cc:2454] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71881b8589eb426982ff85e43eca261b in term 0.
I20260924 17:38:47.261198 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37b87ffb8b04ef2bcc9e47fc464e5a3" candidate_uuid: "71881b8589eb426982ff85e43eca261b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde" is_pre_election: true
I20260924 17:38:47.261260 2344 leader_election.cc:302] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, 71881b8589eb426982ff85e43eca261b; no voters:
I20260924 17:38:47.261343 2553 raft_consensus.cc:2454] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71881b8589eb426982ff85e43eca261b in term 0.
I20260924 17:38:47.261384 2625 raft_consensus.cc:2781] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:47.261416 2625 raft_consensus.cc:482] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:47.261447 2625 raft_consensus.cc:3037] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.262199 2625 raft_consensus.cc:504] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.262355 2625 leader_election.cc:288] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [CANDIDATE]: Term 1 election: Requested vote from peers 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:47.262517 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37b87ffb8b04ef2bcc9e47fc464e5a3" candidate_uuid: "71881b8589eb426982ff85e43eca261b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357"
I20260924 17:38:47.262517 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37b87ffb8b04ef2bcc9e47fc464e5a3" candidate_uuid: "71881b8589eb426982ff85e43eca261b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde"
I20260924 17:38:47.262590 2273 raft_consensus.cc:3037] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.262590 2553 raft_consensus.cc:3037] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.263422 2273 raft_consensus.cc:2454] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71881b8589eb426982ff85e43eca261b in term 1.
I20260924 17:38:47.263422 2553 raft_consensus.cc:2454] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71881b8589eb426982ff85e43eca261b in term 1.
I20260924 17:38:47.263597 2347 leader_election.cc:302] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71881b8589eb426982ff85e43eca261b, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:47.263728 2625 raft_consensus.cc:2781] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:47.263882 2625 raft_consensus.cc:686] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [term 1 LEADER]: Becoming Leader. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Running, Role: LEADER
I20260924 17:38:47.263988 2625 consensus_queue.cc:237] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.264416 2641 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: TxnStatusTablet state changed. Reason: New leader 71881b8589eb426982ff85e43eca261b. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.264515 2641 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:47.264659 2643 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:38:47.264822 2643 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:38:47.264858 2116 catalog_manager.cc:5495] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b reported cstate change: term changed from 0 to 1, leader changed from <none> to 71881b8589eb426982ff85e43eca261b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:38:47.274032 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.271402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.2:40867)
0924 17:38:47.271456 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.271458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.271480 (+ 22us) server_negotiation.cc:407] Connection header received
0924 17:38:47.271555 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.271556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.271573 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.271634 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.271772 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.272259 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.273031 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.273259 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.273339 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.273528 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.273633 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.273882 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.273912 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.273939 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:38:47.274029 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.271347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.0.1:48498)
0924 17:38:47.271429 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.271442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.271480 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.271639 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.271640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.271641 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:47.271698 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.271700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.272352 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.272353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.272943 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.272945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.273338 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.273339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.273406 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.273433 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.273848 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.273862 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.273907 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.273957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 17:38:47.276145 2553 raft_consensus.cc:1261] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Refusing update from remote peer 71881b8589eb426982ff85e43eca261b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.276147 2273 raft_consensus.cc:1261] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Refusing update from remote peer 71881b8589eb426982ff85e43eca261b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.276497 2641 consensus_queue.cc:1035] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.276676 2625 consensus_queue.cc:1035] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.277818 2624 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: TxnStatusTablet state changed. Reason: New leader 71881b8589eb426982ff85e43eca261b. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.277846 2627 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: TxnStatusTablet state changed. Reason: New leader 71881b8589eb426982ff85e43eca261b. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.277916 2624 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.277926 2627 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.278937 2646 mvcc.cc:205] Tried to move back new op lower bound from 7332952175719915520 to 7332952175673888768. Current Snapshot: MvccSnapshot[applied={T|T < 7332952175719915520}]
I20260924 17:38:47.279021 2625 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.279219 2625 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:47.279891 2625 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.279956 2625 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 71881b8589eb426982ff85e43eca261b: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:38:47.280237 2627 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.280311 2627 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P 5b93563c341c4d00bbc600ccf6c91357: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.280783 2624 tablet_replica.cc:439] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71881b8589eb426982ff85e43eca261b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } } }
I20260924 17:38:47.280858 2624 tablet_replica.cc:442] T f37b87ffb8b04ef2bcc9e47fc464e5a3 P b3ff38f71473424aa85221696186efde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:38:47.284483 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.282001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.98.3:43477)
0924 17:38:47.282045 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.282047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.282056 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:47.282130 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.282131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.282147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.282228 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.282373 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.282846 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.283693 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.283903 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.283985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.284170 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.284277 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.284365 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.284387 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.284409 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:38:47.289342 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.286718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.98.3:42053)
0924 17:38:47.286793 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.286804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.286846 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.286966 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.286967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.286967 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:47.287014 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.287015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.287944 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.287945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.288546 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.288548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.288989 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.288990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.289037 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.289049 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.289225 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.289226 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.289262 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.289295 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:38:47.289358 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.286791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42053 (local address 127.0.98.2:40867)
0924 17:38:47.286853 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.286855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.286866 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:38:47.286892 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.286892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.286903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.286965 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.287124 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.287845 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.288625 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.288917 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.288987 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.289133 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.289165 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.289225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.289264 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.289285 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:38:47.292161 2273 raft_consensus.cc:1261] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Refusing update from remote peer b3ff38f71473424aa85221696186efde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.292279 2413 raft_consensus.cc:1261] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Refusing update from remote peer b3ff38f71473424aa85221696186efde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.292412 2629 consensus_queue.cc:1035] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.292510 2624 consensus_queue.cc:1035] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [LEADER]: Connected to new peer: Peer: permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.293751 2648 mvcc.cc:205] Tried to move back new op lower bound from 7332952175786876928 to 7332952175037890560. Current Snapshot: MvccSnapshot[applied={T|T < 7332952175786876928}]
I20260924 17:38:47.293931 2647 mvcc.cc:205] Tried to move back new op lower bound from 7332952175786876928 to 7332952175037890560. Current Snapshot: MvccSnapshot[applied={T|T < 7332952175786876928}]
I20260924 17:38:47.300901 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.298197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48827 (local address 127.0.98.3:43477)
0924 17:38:47.298234 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.298236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.298368 (+ 132us) server_negotiation.cc:407] Connection header received
0924 17:38:47.298427 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.298428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.298438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.298498 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.298650 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.299393 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.300222 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.300423 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.300502 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.300665 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.300697 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.300762 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.300821 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.300839 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:38:47.300948 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.298151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43477 (local address 127.0.98.2:48827)
0924 17:38:47.298303 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.298315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.298376 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.298516 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.298517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.298519 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:47.298581 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.298583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.299461 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.299461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.300160 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.300162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.300504 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.300504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.300569 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.300584 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.300762 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.300764 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.300817 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.300860 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260924 17:38:47.308012 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:47.308169 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:47.308887 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Sending DeleteTablet for 3 replicas of tablet ef6c30ae2ef44f18be8fedff53e287ab
I20260924 17:38:47.309252 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48494
I20260924 17:38:47.309367 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59498
I20260924 17:38:47.309466 2664 tablet_replica.cc:330] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: stopping tablet replica
I20260924 17:38:47.309494 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c30ae2ef44f18be8fedff53e287ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260924 17:38:47.309566 2664 raft_consensus.cc:2229] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:47.309633 2664 raft_consensus.cc:2258] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:47.310182 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:47.310338 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:47.310716 2664 ts_tablet_manager.cc:1905] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:47.311635 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 98969d92f2214954a34be1159c0dfab4 (DEFAULT_TABLE table=test-table [id=1385e154d7ee4b3a83312f4479fc923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:47.311779 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98969d92f2214954a34be1159c0dfab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.312700 2623 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Bootstrap starting.
I20260924 17:38:47.312836 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 98969d92f2214954a34be1159c0dfab4 (DEFAULT_TABLE table=test-table [id=1385e154d7ee4b3a83312f4479fc923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:47.312948 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98969d92f2214954a34be1159c0dfab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.313122 2623 tablet_bootstrap.cc:654] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.313822 2623 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: No bootstrap required, opened a new log
I20260924 17:38:47.313874 2623 ts_tablet_manager.cc:1397] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:47.314007 2623 raft_consensus.cc:348] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.314078 2623 raft_consensus.cc:374] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.314100 2623 raft_consensus.cc:729] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.314139 2623 consensus_queue.cc:260] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.314165 2620 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Bootstrap starting.
I20260924 17:38:47.314252 2623 ts_tablet_manager.cc:1428] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:47.314584 2620 tablet_bootstrap.cc:654] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.315264 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 98969d92f2214954a34be1159c0dfab4 (DEFAULT_TABLE table=test-table [id=1385e154d7ee4b3a83312f4479fc923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:47.315374 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98969d92f2214954a34be1159c0dfab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:47.315392 2620 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: No bootstrap required, opened a new log
I20260924 17:38:47.315428 2620 ts_tablet_manager.cc:1397] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:47.315613 2620 raft_consensus.cc:348] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.315686 2620 raft_consensus.cc:374] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.315706 2620 raft_consensus.cc:729] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ff38f71473424aa85221696186efde, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.315745 2620 consensus_queue.cc:260] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.315820 2620 ts_tablet_manager.cc:1428] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:47.316771 2619 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Bootstrap starting.
I20260924 17:38:47.317149 2619 tablet_bootstrap.cc:654] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:47.317713 2619 tablet_bootstrap.cc:492] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: No bootstrap required, opened a new log
I20260924 17:38:47.317776 2619 ts_tablet_manager.cc:1397] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:47.317901 2665 tablet_replica.cc:330] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: stopping tablet replica
I20260924 17:38:47.317915 2619 raft_consensus.cc:348] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.317997 2619 raft_consensus.cc:374] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:47.318038 2619 raft_consensus.cc:729] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Initialized, Role: FOLLOWER
I20260924 17:38:47.318034 2665 raft_consensus.cc:2229] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:47.318090 2619 consensus_queue.cc:260] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.318143 2665 raft_consensus.cc:2258] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:47.318192 2619 ts_tablet_manager.cc:1428] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:47.319139 2666 tablet_replica.cc:330] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: stopping tablet replica
I20260924 17:38:47.319274 2666 raft_consensus.cc:2229] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:47.319337 2666 raft_consensus.cc:2258] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:47.319495 2665 ts_tablet_manager.cc:1905] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:47.319743 2666 ts_tablet_manager.cc:1905] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:47.319813 2664 ts_tablet_manager.cc:1918] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:47.319892 2664 log.cc:1196] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/ef6c30ae2ef44f18be8fedff53e287ab
I20260924 17:38:47.320182 2664 ts_tablet_manager.cc:1939] T ef6c30ae2ef44f18be8fedff53e287ab P 71881b8589eb426982ff85e43eca261b: Deleting consensus metadata
I20260924 17:38:47.320640 2101 catalog_manager.cc:4841] TS 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867): tablet ef6c30ae2ef44f18be8fedff53e287ab (table test-table [id=4713342818df46f38cbca308b4d49c09]) successfully deleted
I20260924 17:38:47.321408 2666 ts_tablet_manager.cc:1918] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:47.321482 2666 log.cc:1196] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/ef6c30ae2ef44f18be8fedff53e287ab
I20260924 17:38:47.321575 2665 ts_tablet_manager.cc:1918] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:47.321668 2665 log.cc:1196] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/ef6c30ae2ef44f18be8fedff53e287ab
I20260924 17:38:47.321782 2666 ts_tablet_manager.cc:1939] T ef6c30ae2ef44f18be8fedff53e287ab P 5b93563c341c4d00bbc600ccf6c91357: Deleting consensus metadata
I20260924 17:38:47.321956 2665 ts_tablet_manager.cc:1939] T ef6c30ae2ef44f18be8fedff53e287ab P b3ff38f71473424aa85221696186efde: Deleting consensus metadata
I20260924 17:38:47.322202 2100 catalog_manager.cc:4841] TS 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375): tablet ef6c30ae2ef44f18be8fedff53e287ab (table test-table [id=4713342818df46f38cbca308b4d49c09]) successfully deleted
I20260924 17:38:47.322342 2103 catalog_manager.cc:4841] TS b3ff38f71473424aa85221696186efde (127.0.98.3:43477): tablet ef6c30ae2ef44f18be8fedff53e287ab (table test-table [id=4713342818df46f38cbca308b4d49c09]) successfully deleted
I20260924 17:38:47.537196 2627 raft_consensus.cc:482] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:47.537318 2627 raft_consensus.cc:504] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.537832 2627 leader_election.cc:288] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:47.541863 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.538122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47749 (local address 127.0.98.3:43477)
0924 17:38:47.538169 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.538171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.538310 (+ 139us) server_negotiation.cc:407] Connection header received
0924 17:38:47.538381 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.538382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.538391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.538450 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.538634 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.539407 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.540976 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.541484 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.541540 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.541548 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.541735 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.541796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:38:47.541952 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.538125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59997 (local address 127.0.98.2:40867)
0924 17:38:47.538212 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.538214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.538223 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:47.538286 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.538287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.538295 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.538368 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:47.538639 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.539407 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.541019 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.541553 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.541622 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.541633 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.541827 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.541871 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 17:38:47.541989 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.538000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.98.1:59997)
0924 17:38:47.538141 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.538154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.538216 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.538442 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.538443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.538445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.538551 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.538552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.539485 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.539486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.540874 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.540878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.541643 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.541644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.541743 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.541748 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.541807 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.541850 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":64,"thread_start_us":61,"threads_started":1}
I20260924 17:38:47.541867 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.538020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43477 (local address 127.0.98.1:47749)
0924 17:38:47.538245 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.538256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.538324 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.538459 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.538459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.538460 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:47.538550 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.538552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.539485 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.539486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.540874 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.540878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.541562 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.541570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.541652 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.541657 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.541709 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.541746 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":85,"thread_start_us":53,"threads_started":1}
I20260924 17:38:47.542261 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98969d92f2214954a34be1159c0dfab4" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b" is_pre_election: true
I20260924 17:38:47.542373 2413 raft_consensus.cc:2454] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 0.
I20260924 17:38:47.542405 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98969d92f2214954a34be1159c0dfab4" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde" is_pre_election: true
I20260924 17:38:47.542497 2553 raft_consensus.cc:2454] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 0.
I20260924 17:38:47.542554 2205 leader_election.cc:302] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, 71881b8589eb426982ff85e43eca261b; no voters:
I20260924 17:38:47.542757 2627 raft_consensus.cc:2781] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:47.542804 2627 raft_consensus.cc:482] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:47.542836 2627 raft_consensus.cc:3037] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.543639 2627 raft_consensus.cc:504] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.543807 2627 leader_election.cc:288] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 election: Requested vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:47.543931 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98969d92f2214954a34be1159c0dfab4" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b"
I20260924 17:38:47.543970 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98969d92f2214954a34be1159c0dfab4" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde"
I20260924 17:38:47.544011 2413 raft_consensus.cc:3037] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.544037 2553 raft_consensus.cc:3037] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:47.544747 2413 raft_consensus.cc:2454] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 1.
I20260924 17:38:47.544747 2553 raft_consensus.cc:2454] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 1.
I20260924 17:38:47.544948 2207 leader_election.cc:302] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:47.545039 2627 raft_consensus.cc:2781] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:47.545166 2627 raft_consensus.cc:686] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 LEADER]: Becoming Leader. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Running, Role: LEADER
I20260924 17:38:47.545223 2627 consensus_queue.cc:237] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:47.545982 2115 catalog_manager.cc:5495] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5b93563c341c4d00bbc600ccf6c91357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:38:47.602943 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.600876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.0.98.1:43375)
0924 17:38:47.600918 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.600920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.600927 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:38:47.600975 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.600976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.600994 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.601062 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.601144 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.601605 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.602298 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.602476 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.602529 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.602659 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.602765 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.602822 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.602867 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.602884 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:38:47.607769 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.605299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.98.1:51753)
0924 17:38:47.605441 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.605451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.605514 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.605633 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.605633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.605635 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:47.605697 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.605698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.606326 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.606328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.606919 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.606921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.607309 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.607310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.607374 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.607397 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.607583 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.607585 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.607627 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.607673 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 17:38:47.607743 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.605373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51753 (local address 127.0.98.2:40867)
0924 17:38:47.605422 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.605424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.605518 (+ 94us) server_negotiation.cc:407] Connection header received
0924 17:38:47.605565 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.605566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.605577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.605621 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.605785 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.606241 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.607010 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.607228 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.607308 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.607478 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.607517 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.607581 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.607638 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.607658 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:38:47.609804 2413 raft_consensus.cc:1261] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Refusing update from remote peer 5b93563c341c4d00bbc600ccf6c91357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.609836 2553 raft_consensus.cc:1261] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Refusing update from remote peer 5b93563c341c4d00bbc600ccf6c91357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:47.610272 2672 consensus_queue.cc:1035] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.610430 2627 consensus_queue.cc:1035] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:47.617329 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.614996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45611 (local address 127.0.98.1:43375)
0924 17:38:47.615077 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:47.615079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:47.615088 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:47.615108 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:47.615109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:47.615120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:47.615184 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:47.615329 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.615833 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.616636 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.616918 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.616990 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.617105 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.617138 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.617186 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:47.617246 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 17:38:47.617269 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 17:38:47.617334 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:47.614916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43375 (local address 127.0.98.2:45611)
0924 17:38:47.614953 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:47.614964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:47.615002 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:47.615217 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:47.615217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:47.615219 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:38:47.615267 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.615269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.615906 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.615907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.616514 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:47.616516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:47.616969 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:47.616969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:47.617030 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:47.617037 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:38:47.617187 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:38:47.617188 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:38:47.617238 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:38:47.617280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 17:38:47.617666 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 17:38:49.139811 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:49.139945 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:49.140740 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Sending DeleteTablet for 3 replicas of tablet 98969d92f2214954a34be1159c0dfab4
I20260924 17:38:49.140980 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 98969d92f2214954a34be1159c0dfab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48494
I20260924 17:38:49.140969 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 98969d92f2214954a34be1159c0dfab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260924 17:38:49.141012 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 98969d92f2214954a34be1159c0dfab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59498
I20260924 17:38:49.141278 2704 tablet_replica.cc:330] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: stopping tablet replica
I20260924 17:38:49.141340 2704 raft_consensus.cc:2229] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:49.141377 2704 raft_consensus.cc:2258] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.141497 2703 tablet_replica.cc:330] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: stopping tablet replica
I20260924 17:38:49.141564 2703 raft_consensus.cc:2229] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:49.141610 2703 raft_consensus.cc:2258] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.141821 2704 ts_tablet_manager.cc:1905] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.143782 2704 ts_tablet_manager.cc:1918] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:38:49.143862 2704 log.cc:1196] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/98969d92f2214954a34be1159c0dfab4
I20260924 17:38:49.144076 2704 ts_tablet_manager.cc:1939] T 98969d92f2214954a34be1159c0dfab4 P b3ff38f71473424aa85221696186efde: Deleting consensus metadata
I20260924 17:38:49.144263 2702 tablet_replica.cc:330] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: stopping tablet replica
I20260924 17:38:49.144330 2702 raft_consensus.cc:2229] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:49.144436 2702 raft_consensus.cc:2258] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.144635 2703 ts_tablet_manager.cc:1905] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.144793 2103 catalog_manager.cc:4841] TS b3ff38f71473424aa85221696186efde (127.0.98.3:43477): tablet 98969d92f2214954a34be1159c0dfab4 (table test-table [id=1385e154d7ee4b3a83312f4479fc923e]) successfully deleted
I20260924 17:38:49.146500 2703 ts_tablet_manager.cc:1918] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:38:49.146553 2703 log.cc:1196] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/98969d92f2214954a34be1159c0dfab4
I20260924 17:38:49.146773 2703 ts_tablet_manager.cc:1939] T 98969d92f2214954a34be1159c0dfab4 P 71881b8589eb426982ff85e43eca261b: Deleting consensus metadata
I20260924 17:38:49.147121 2702 ts_tablet_manager.cc:1905] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.147397 2101 catalog_manager.cc:4841] TS 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867): tablet 98969d92f2214954a34be1159c0dfab4 (table test-table [id=1385e154d7ee4b3a83312f4479fc923e]) successfully deleted
I20260924 17:38:49.148641 2702 ts_tablet_manager.cc:1918] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:38:49.148703 2702 log.cc:1196] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/98969d92f2214954a34be1159c0dfab4
I20260924 17:38:49.148927 2702 ts_tablet_manager.cc:1939] T 98969d92f2214954a34be1159c0dfab4 P 5b93563c341c4d00bbc600ccf6c91357: Deleting consensus metadata
I20260924 17:38:49.149345 2100 catalog_manager.cc:4841] TS 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375): tablet 98969d92f2214954a34be1159c0dfab4 (table test-table [id=1385e154d7ee4b3a83312f4479fc923e]) successfully deleted
Sep 24 17:38:49 dist-test-slave-d0pf krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:38:49 dist-test-slave-d0pf krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:49.170660 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:49.158512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.0.98.62:39635)
0924 17:38:49.158658 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:49.158660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:49.158733 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:38:49.158835 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:49.158836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:49.158855 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:49.158926 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:49.159563 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.160090 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.160854 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.161057 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.161133 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:49.167797 (+ 6664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:49.167798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:49.167799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:49.167814 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:49.169463 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:49.169819 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:49.169820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:49.169820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:49.169883 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:49.170093 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:49.170094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:49.170094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:49.170213 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:49.170321 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:49.170441 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 17:38:49.170529 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 17:38:49.172066 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:49.172221 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:49.173604 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb (DEFAULT_TABLE table=test-table [id=a63168e946844c75b7d63c154dcec028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.173621 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb (DEFAULT_TABLE table=test-table [id=a63168e946844c75b7d63c154dcec028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.173606 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb (DEFAULT_TABLE table=test-table [id=a63168e946844c75b7d63c154dcec028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.173753 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b323ea4fa9e46d0850a90687330e7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.173754 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b323ea4fa9e46d0850a90687330e7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.173764 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b323ea4fa9e46d0850a90687330e7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.174803 2713 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Bootstrap starting.
I20260924 17:38:49.175091 2714 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Bootstrap starting.
I20260924 17:38:49.175285 2713 tablet_bootstrap.cc:654] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.175450 2714 tablet_bootstrap.cc:654] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.175740 2715 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Bootstrap starting.
I20260924 17:38:49.176014 2714 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: No bootstrap required, opened a new log
I20260924 17:38:49.176038 2713 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: No bootstrap required, opened a new log
I20260924 17:38:49.176082 2713 ts_tablet_manager.cc:1397] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.176083 2714 ts_tablet_manager.cc:1397] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.176100 2715 tablet_bootstrap.cc:654] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.176198 2714 raft_consensus.cc:348] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.176291 2714 raft_consensus.cc:374] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.176314 2714 raft_consensus.cc:729] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.176277 2713 raft_consensus.cc:348] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.176384 2713 raft_consensus.cc:374] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.176378 2714 consensus_queue.cc:260] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.176432 2713 raft_consensus.cc:729] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.176484 2714 ts_tablet_manager.cc:1428] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.176525 2713 consensus_queue.cc:260] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.176631 2713 ts_tablet_manager.cc:1428] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.177034 2715 tablet_bootstrap.cc:492] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: No bootstrap required, opened a new log
I20260924 17:38:49.177130 2715 ts_tablet_manager.cc:1397] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.177286 2715 raft_consensus.cc:348] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.177388 2715 raft_consensus.cc:374] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.177436 2715 raft_consensus.cc:729] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ff38f71473424aa85221696186efde, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.177491 2715 consensus_queue.cc:260] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.177654 2715 ts_tablet_manager.cc:1428] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.297998 2719 raft_consensus.cc:482] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:49.298094 2719 raft_consensus.cc:504] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.298256 2719 leader_election.cc:288] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375), 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867)
I20260924 17:38:49.298472 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b323ea4fa9e46d0850a90687330e7bb" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357" is_pre_election: true
I20260924 17:38:49.298496 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b323ea4fa9e46d0850a90687330e7bb" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b" is_pre_election: true
I20260924 17:38:49.298578 2273 raft_consensus.cc:2454] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 0.
I20260924 17:38:49.298607 2413 raft_consensus.cc:2454] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 0.
I20260924 17:38:49.298775 2485 leader_election.cc:302] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71881b8589eb426982ff85e43eca261b, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:49.298893 2719 raft_consensus.cc:2781] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:49.298964 2719 raft_consensus.cc:482] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:49.298979 2719 raft_consensus.cc:3037] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.299708 2719 raft_consensus.cc:504] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.299840 2719 leader_election.cc:288] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 election: Requested vote from peers 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375), 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867)
I20260924 17:38:49.299998 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b323ea4fa9e46d0850a90687330e7bb" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b"
I20260924 17:38:49.300096 2413 raft_consensus.cc:3037] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.300004 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b323ea4fa9e46d0850a90687330e7bb" candidate_uuid: "b3ff38f71473424aa85221696186efde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93563c341c4d00bbc600ccf6c91357"
I20260924 17:38:49.300154 2273 raft_consensus.cc:3037] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.300782 2273 raft_consensus.cc:2454] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 1.
I20260924 17:38:49.300854 2413 raft_consensus.cc:2454] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ff38f71473424aa85221696186efde in term 1.
I20260924 17:38:49.300985 2484 leader_election.cc:302] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, b3ff38f71473424aa85221696186efde; no voters:
I20260924 17:38:49.301075 2719 raft_consensus.cc:2781] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:49.301210 2719 raft_consensus.cc:686] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 1 LEADER]: Becoming Leader. State: Replica: b3ff38f71473424aa85221696186efde, State: Running, Role: LEADER
I20260924 17:38:49.301255 2719 consensus_queue.cc:237] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.301885 2114 catalog_manager.cc:5495] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ff38f71473424aa85221696186efde (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b3ff38f71473424aa85221696186efde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } health_report { overall_health: HEALTHY } } }
I20260924 17:38:49.327720 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:49.325419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.0.98.3:43477)
0924 17:38:49.325576 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:49.325577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:49.325585 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:49.325613 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:49.325613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:49.325623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:49.325686 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:49.325781 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.326245 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.326975 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.327252 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.327316 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:49.327499 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:49.327531 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:49.327583 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:49.327636 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 17:38:49.327656 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260924 17:38:49.330076 2273 raft_consensus.cc:1261] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Refusing update from remote peer b3ff38f71473424aa85221696186efde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:49.330065 2413 raft_consensus.cc:1261] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Refusing update from remote peer b3ff38f71473424aa85221696186efde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:49.330293 2720 consensus_queue.cc:1035] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:49.330399 2719 consensus_queue.cc:1035] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [LEADER]: Connected to new peer: Peer: permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:49.340390 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:49.340490 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:49.341048 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Sending DeleteTablet for 3 replicas of tablet 4b323ea4fa9e46d0850a90687330e7bb
I20260924 17:38:49.341269 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48494
I20260924 17:38:49.341274 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260924 17:38:49.341368 2703 tablet_replica.cc:330] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: stopping tablet replica
I20260924 17:38:49.341369 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b323ea4fa9e46d0850a90687330e7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59498
I20260924 17:38:49.341426 2703 raft_consensus.cc:2229] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:49.341454 2704 tablet_replica.cc:330] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: stopping tablet replica
I20260924 17:38:49.341456 2702 tablet_replica.cc:330] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: stopping tablet replica
I20260924 17:38:49.341470 2703 raft_consensus.cc:2258] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.341506 2704 raft_consensus.cc:2229] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:49.341521 2702 raft_consensus.cc:2229] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:49.341552 2702 raft_consensus.cc:2258] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.341562 2704 raft_consensus.cc:2258] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:49.341548 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:49.341712 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:49.341858 2702 ts_tablet_manager.cc:1905] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.342005 2703 ts_tablet_manager.cc:1905] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.342032 2704 ts_tablet_manager.cc:1905] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:49.343247 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f (DEFAULT_TABLE table=test-table [id=dca091e17c7c4d1cb7ef19f4fbb0fe93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.343248 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f (DEFAULT_TABLE table=test-table [id=dca091e17c7c4d1cb7ef19f4fbb0fe93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.343345 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d43f484c9ae49fcba876d7400a35e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.343353 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d43f484c9ae49fcba876d7400a35e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.343539 2703 ts_tablet_manager.cc:1918] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:49.343586 2703 log.cc:1196] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/4b323ea4fa9e46d0850a90687330e7bb
I20260924 17:38:49.343776 2704 ts_tablet_manager.cc:1918] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:49.343798 2703 ts_tablet_manager.cc:1939] T 4b323ea4fa9e46d0850a90687330e7bb P 71881b8589eb426982ff85e43eca261b: Deleting consensus metadata
I20260924 17:38:49.343824 2704 log.cc:1196] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/4b323ea4fa9e46d0850a90687330e7bb
I20260924 17:38:49.344031 2704 ts_tablet_manager.cc:1939] T 4b323ea4fa9e46d0850a90687330e7bb P b3ff38f71473424aa85221696186efde: Deleting consensus metadata
I20260924 17:38:49.344272 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f (DEFAULT_TABLE table=test-table [id=dca091e17c7c4d1cb7ef19f4fbb0fe93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:49.344314 2101 catalog_manager.cc:4841] TS 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867): tablet 4b323ea4fa9e46d0850a90687330e7bb (table test-table [id=a63168e946844c75b7d63c154dcec028]) successfully deleted
I20260924 17:38:49.344358 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d43f484c9ae49fcba876d7400a35e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:49.344669 2715 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Bootstrap starting.
I20260924 17:38:49.344707 2714 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357: Bootstrap starting.
I20260924 17:38:49.345006 2714 tablet_bootstrap.cc:654] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.345007 2715 tablet_bootstrap.cc:654] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.345513 2103 catalog_manager.cc:4841] TS b3ff38f71473424aa85221696186efde (127.0.98.3:43477): tablet 4b323ea4fa9e46d0850a90687330e7bb (table test-table [id=a63168e946844c75b7d63c154dcec028]) successfully deleted
I20260924 17:38:49.345618 2713 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Bootstrap starting.
I20260924 17:38:49.345646 2714 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357: No bootstrap required, opened a new log
I20260924 17:38:49.345683 2714 ts_tablet_manager.cc:1397] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.345825 2715 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: No bootstrap required, opened a new log
I20260924 17:38:49.345860 2715 ts_tablet_manager.cc:1397] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.345815 2714 raft_consensus.cc:348] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.345889 2714 raft_consensus.cc:374] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.345916 2714 raft_consensus.cc:729] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.345938 2713 tablet_bootstrap.cc:654] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:49.345932 2715 raft_consensus.cc:348] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.345974 2715 raft_consensus.cc:374] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.345990 2715 raft_consensus.cc:729] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ff38f71473424aa85221696186efde, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.345963 2714 consensus_queue.cc:260] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.346051 2715 consensus_queue.cc:260] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.346068 2714 ts_tablet_manager.cc:1428] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.346143 2715 ts_tablet_manager.cc:1428] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.346715 2713 tablet_bootstrap.cc:492] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: No bootstrap required, opened a new log
I20260924 17:38:49.346746 2702 ts_tablet_manager.cc:1918] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:38:49.346784 2713 ts_tablet_manager.cc:1397] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:49.346784 2702 log.cc:1196] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/4b323ea4fa9e46d0850a90687330e7bb
I20260924 17:38:49.346925 2713 raft_consensus.cc:348] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.347000 2713 raft_consensus.cc:374] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:49.346997 2702 ts_tablet_manager.cc:1939] T 4b323ea4fa9e46d0850a90687330e7bb P 5b93563c341c4d00bbc600ccf6c91357: Deleting consensus metadata
I20260924 17:38:49.347034 2713 raft_consensus.cc:729] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71881b8589eb426982ff85e43eca261b, State: Initialized, Role: FOLLOWER
I20260924 17:38:49.347110 2713 consensus_queue.cc:260] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.347206 2713 ts_tablet_manager.cc:1428] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:49.347469 2100 catalog_manager.cc:4841] TS 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375): tablet 4b323ea4fa9e46d0850a90687330e7bb (table test-table [id=a63168e946844c75b7d63c154dcec028]) successfully deleted
I20260924 17:38:49.557816 2681 raft_consensus.cc:482] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:49.557927 2681 raft_consensus.cc:504] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.558194 2681 leader_election.cc:288] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:49.558429 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d43f484c9ae49fcba876d7400a35e2f" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde" is_pre_election: true
I20260924 17:38:49.558455 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d43f484c9ae49fcba876d7400a35e2f" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b" is_pre_election: true
I20260924 17:38:49.558531 2413 raft_consensus.cc:2454] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 0.
I20260924 17:38:49.558550 2553 raft_consensus.cc:2454] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 0.
I20260924 17:38:49.558684 2205 leader_election.cc:302] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, 71881b8589eb426982ff85e43eca261b; no voters:
I20260924 17:38:49.558799 2681 raft_consensus.cc:2781] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:49.558849 2681 raft_consensus.cc:482] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:49.558877 2681 raft_consensus.cc:3037] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.559624 2681 raft_consensus.cc:504] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.559767 2681 leader_election.cc:288] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 election: Requested vote from peers 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867), b3ff38f71473424aa85221696186efde (127.0.98.3:43477)
I20260924 17:38:49.559882 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d43f484c9ae49fcba876d7400a35e2f" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71881b8589eb426982ff85e43eca261b"
I20260924 17:38:49.559976 2413 raft_consensus.cc:3037] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.559917 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d43f484c9ae49fcba876d7400a35e2f" candidate_uuid: "5b93563c341c4d00bbc600ccf6c91357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ff38f71473424aa85221696186efde"
I20260924 17:38:49.560045 2553 raft_consensus.cc:3037] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:49.560535 2413 raft_consensus.cc:2454] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 1.
I20260924 17:38:49.560678 2553 raft_consensus.cc:2454] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93563c341c4d00bbc600ccf6c91357 in term 1.
I20260924 17:38:49.560702 2205 leader_election.cc:302] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93563c341c4d00bbc600ccf6c91357, 71881b8589eb426982ff85e43eca261b; no voters:
I20260924 17:38:49.560778 2681 raft_consensus.cc:2781] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:49.560822 2681 raft_consensus.cc:686] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [term 1 LEADER]: Becoming Leader. State: Replica: 5b93563c341c4d00bbc600ccf6c91357, State: Running, Role: LEADER
I20260924 17:38:49.560871 2681 consensus_queue.cc:237] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } }
I20260924 17:38:49.561468 2114 catalog_manager.cc:5495] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5b93563c341c4d00bbc600ccf6c91357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93563c341c4d00bbc600ccf6c91357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:38:49.575987 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:49.573763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.0.98.1:43375)
0924 17:38:49.573925 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:49.573927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:49.573935 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:49.573964 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:49.573965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:49.573975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:49.574034 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:49.574153 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.574646 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.575423 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:49.575625 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:49.575687 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:49.575797 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:49.575822 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:49.575857 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:49.575908 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 17:38:49.575923 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260924 17:38:49.578172 2553 raft_consensus.cc:1261] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Refusing update from remote peer 5b93563c341c4d00bbc600ccf6c91357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:49.578172 2413 raft_consensus.cc:1261] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Refusing update from remote peer 5b93563c341c4d00bbc600ccf6c91357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:49.578378 2681 consensus_queue.cc:1035] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ff38f71473424aa85221696186efde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:49.578510 2694 consensus_queue.cc:1035] T 3d43f484c9ae49fcba876d7400a35e2f P 5b93563c341c4d00bbc600ccf6c91357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71881b8589eb426982ff85e43eca261b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:38:49.582741 2685 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 17:38:51.099859 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:51.099964 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:51.100855 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9d05c6073b4beabf8b6f9fb70445b5: Sending DeleteTablet for 3 replicas of tablet 3d43f484c9ae49fcba876d7400a35e2f
I20260924 17:38:51.101099 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260924 17:38:51.101224 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59498
I20260924 17:38:51.101300 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d43f484c9ae49fcba876d7400a35e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48494
I20260924 17:38:51.101342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2188
I20260924 17:38:51.102661 2761 tablet_replica.cc:330] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: stopping tablet replica
I20260924 17:38:51.102747 2761 raft_consensus.cc:2229] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:51.102793 2761 raft_consensus.cc:2258] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:51.103101 2761 ts_tablet_manager.cc:1905] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:51.104452 2760 tablet_replica.cc:330] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: stopping tablet replica
I20260924 17:38:51.104537 2760 raft_consensus.cc:2229] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:51.104569 2760 raft_consensus.cc:2258] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:51.104842 2761 ts_tablet_manager.cc:1918] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:38:51.104866 2760 ts_tablet_manager.cc:1905] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:51.104915 2761 log.cc:1196] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/3d43f484c9ae49fcba876d7400a35e2f
I20260924 17:38:51.105191 2761 ts_tablet_manager.cc:1939] T 3d43f484c9ae49fcba876d7400a35e2f P 71881b8589eb426982ff85e43eca261b: Deleting consensus metadata
I20260924 17:38:51.105578 2101 catalog_manager.cc:4841] TS 71881b8589eb426982ff85e43eca261b (127.0.98.2:40867): tablet 3d43f484c9ae49fcba876d7400a35e2f (table test-table [id=dca091e17c7c4d1cb7ef19f4fbb0fe93]) successfully deleted
I20260924 17:38:51.106469 2760 ts_tablet_manager.cc:1918] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:38:51.106520 2760 log.cc:1196] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/3d43f484c9ae49fcba876d7400a35e2f
I20260924 17:38:51.106745 2760 ts_tablet_manager.cc:1939] T 3d43f484c9ae49fcba876d7400a35e2f P b3ff38f71473424aa85221696186efde: Deleting consensus metadata
I20260924 17:38:51.107106 2103 catalog_manager.cc:4841] TS b3ff38f71473424aa85221696186efde (127.0.98.3:43477): tablet 3d43f484c9ae49fcba876d7400a35e2f (table test-table [id=dca091e17c7c4d1cb7ef19f4fbb0fe93]) successfully deleted
W20260924 17:38:51.109366 2100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43375 (error 108)
I20260924 17:38:51.109611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2328
I20260924 17:38:51.110504 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:51.109743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43375 (local address 127.0.0.1:36604)
0924 17:38:51.110402 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:51.110452 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":52,"threads_started":1}
W20260924 17:38:51.110608 2100 catalog_manager.cc:4584] TS 5b93563c341c4d00bbc600ccf6c91357 (127.0.98.1:43375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d43f484c9ae49fcba876d7400a35e2f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43375: connect: Connection refused (error 111)
I20260924 17:38:51.115381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2468
I20260924 17:38:51.120836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2084
2026-09-24T17:38:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5964 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:51 dist-test-slave-d0pf krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:51 dist-test-slave-d0pf krb5kdc[2766](info): set up 2 sockets
Sep 24 17:38:51 dist-test-slave-d0pf krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260924 17:38:52.162881 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:52.282541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:52.363128 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:52.363373 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:52.363408 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:52.364570 2779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:52.364638 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:52.364662 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:52.364681 2779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:52.364701 2779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:52.366205 2779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:52.366478 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:52.366698 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:52.371881 2784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.371881 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.371958 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:52.372010 2779 server_base.cc:1034] running on GCE node
I20260924 17:38:52.372401 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:52.372695 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:52.373849 2779 hybrid_clock.cc:648] HybridClock initialized: now 1790271532373835 us; error 30 us; skew 500 ppm
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:52.376446 2779 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:52.377159 2779 webserver.cc:460] Webserver started at http://127.0.98.62:35621/ using document root <none> and password file <none>
I20260924 17:38:52.377398 2779 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:52.377466 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:52.377619 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:52.378701 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "4383a3dbe0f446a7b961e03fc5080765"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "ae7a5263d24bcfb6d07d2b6350734400"
server_key_iv: "4467fe0d5dae680fc06c7596ca3980bc"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.379135 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "4383a3dbe0f446a7b961e03fc5080765"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "ae7a5263d24bcfb6d07d2b6350734400"
server_key_iv: "4467fe0d5dae680fc06c7596ca3980bc"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.380491 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:52.381233 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.381415 2779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.381513 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "4383a3dbe0f446a7b961e03fc5080765"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "ae7a5263d24bcfb6d07d2b6350734400"
server_key_iv: "4467fe0d5dae680fc06c7596ca3980bc"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.381599 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:52.397367 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:52.399284 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:52.399467 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:52.404129 2779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43937
I20260924 17:38:52.404151 2846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43937 every 8 connection(s)
I20260924 17:38:52.404745 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:52.405510 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:52.407415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2779
I20260924 17:38:52.407513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:52.407678 392 external_mini_cluster.cc:1428] Setting key 84507849f861e59cfa5701497a596e2a
I20260924 17:38:52.407908 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765: Bootstrap starting.
I20260924 17:38:52.408622 2847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:52.408941 2847 log.cc:824] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:52.409651 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765: No bootstrap required, opened a new log
I20260924 17:38:52.411985 2847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260924 17:38:52.412117 2847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:52.412139 2847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4383a3dbe0f446a7b961e03fc5080765, State: Initialized, Role: FOLLOWER
I20260924 17:38:52.412282 2847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260924 17:38:52.412362 2847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:52.412385 2847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:52.412452 2847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:52.413007 2847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260924 17:38:52.413125 2847 leader_election.cc:302] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4383a3dbe0f446a7b961e03fc5080765; no voters:
I20260924 17:38:52.413300 2847 leader_election.cc:288] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:52.413471 2853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:52.413571 2847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:52.413630 2853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [term 1 LEADER]: Becoming Leader. State: Replica: 4383a3dbe0f446a7b961e03fc5080765, State: Running, Role: LEADER
I20260924 17:38:52.413770 2853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260924 17:38:52.414192 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4383a3dbe0f446a7b961e03fc5080765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } } }
I20260924 17:38:52.414273 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:52.414448 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4383a3dbe0f446a7b961e03fc5080765. Latest consensus state: current_term: 1 leader_uuid: "4383a3dbe0f446a7b961e03fc5080765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4383a3dbe0f446a7b961e03fc5080765" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } } }
I20260924 17:38:52.414511 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:52.414805 2861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:52.415465 2861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:52.417008 2861 catalog_manager.cc:1347] Generated new cluster ID: 3e74cfcff1c64d268c1ba49ac9585539
I20260924 17:38:52.417063 2861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:38:52.419466 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.408399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.0.98.62:43937)
0924 17:38:52.408575 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:52.408585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:52.408620 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:38:52.409212 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:52.409226 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:52.409276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:52.409436 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:52.410043 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.411463 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.412230 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.412776 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.412900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.415765 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:52.415768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:52.415769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:52.415780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:52.417850 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.418231 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.418234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.418234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.418340 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.418542 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.418542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.418543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.418857 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:52.419014 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:52.419117 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 17:38:52.419254 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260924 17:38:52.422557 2861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:52.423265 2861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:52.428037 2861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4383a3dbe0f446a7b961e03fc5080765: Generated new TSK 0
I20260924 17:38:52.428218 2861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:52.454663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:52.534786 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:52.535020 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:52.535079 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:52.536276 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:52.536345 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:52.536391 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:52.537901 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:52.538161 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:52.538388 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:52.543316 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.543356 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:52.543426 2873 server_base.cc:1034] running on GCE node
W20260924 17:38:52.543316 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:52.543804 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:52.544067 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:52.545209 2873 hybrid_clock.cc:648] HybridClock initialized: now 1790271532545197 us; error 29 us; skew 500 ppm
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:52.547645 2873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:52.548089 2873 webserver.cc:460] Webserver started at http://127.0.98.1:32929/ using document root <none> and password file <none>
I20260924 17:38:52.548300 2873 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:52.548342 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:52.548431 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:52.549410 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec4651ffcde6455e857aa9dafd6719a1"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "409cdd7f7b604faeb1e5bd540db680d6"
server_key_iv: "82da40cf0faa01a59051273dc3a7c38c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.549708 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec4651ffcde6455e857aa9dafd6719a1"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "409cdd7f7b604faeb1e5bd540db680d6"
server_key_iv: "82da40cf0faa01a59051273dc3a7c38c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.550925 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:52.551697 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.551887 2873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:52.551973 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "ec4651ffcde6455e857aa9dafd6719a1"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "409cdd7f7b604faeb1e5bd540db680d6"
server_key_iv: "82da40cf0faa01a59051273dc3a7c38c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.552026 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:52.564713 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:52.566475 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:52.566601 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:52.566774 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:52.567140 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:52.567183 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.567205 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:52.567248 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.572947 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44309
I20260924 17:38:52.572990 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44309 every 8 connection(s)
I20260924 17:38:52.573335 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:52.579624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 2873
I20260924 17:38:52.579731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:52.579944 392 external_mini_cluster.cc:1428] Setting key 6ab6f755514a65849bcf977e279caafc
I20260924 17:38:52.582942 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.574100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44979 (local address 127.0.98.62:43937)
0924 17:38:52.574157 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:52.574159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:52.574363 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:38:52.575379 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:52.575380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:52.575391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:52.575439 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:52.576022 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.576548 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.577527 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.577738 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.577811 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.579940 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:52.579942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:52.579942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:52.579950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:52.581546 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.581932 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.581933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.581934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.582004 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.582311 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.582312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.582312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.582435 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:52.582488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:52.582812 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 17:38:52.582877 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:38:52.583287 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.574073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43937 (local address 127.0.98.1:44979)
0924 17:38:52.574235 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:52.574271 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:52.575285 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:52.575502 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:52.575505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:52.575749 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:52.575946 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.575951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.576626 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.576627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.577413 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.577415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.577830 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.577831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.577956 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.578404 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:52.578412 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:52.579841 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:52.581626 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.581631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.581633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.581856 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.582085 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.582086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.582087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.582243 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.582501 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:52.582507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:52.582674 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 17:38:52.582794 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 17:38:52.582981 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260924 17:38:52.583880 3002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43937
I20260924 17:38:52.583982 3002 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:52.584172 3002 heartbeater.cc:511] Master 127.0.98.62:43937 requested a full tablet report, sending...
I20260924 17:38:52.584694 2811 ts_manager.cc:194] Registered new tserver with Master: ec4651ffcde6455e857aa9dafd6719a1 (127.0.98.1:44309)
I20260924 17:38:52.585289 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44979
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:52.605734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:52.686050 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:52.686255 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:52.686277 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:52.687415 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:52.687454 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:52.687480 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:52.688913 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:52.689083 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:52.689260 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:52.693989 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.694108 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.694191 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:52.694216 3007 server_base.cc:1034] running on GCE node
I20260924 17:38:52.694433 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:52.694710 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:52.695855 3007 hybrid_clock.cc:648] HybridClock initialized: now 1790271532695845 us; error 27 us; skew 500 ppm
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:52.698244 3007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:52.698698 3007 webserver.cc:460] Webserver started at http://127.0.98.2:33803/ using document root <none> and password file <none>
I20260924 17:38:52.698918 3007 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:52.698987 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:52.699156 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:52.700263 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ad8d609029540708c46281d484f09fb"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "caea743c1026e75c70bcd90a5ac830a6"
server_key_iv: "7920e223ab44b96b92521f6c6e74a4be"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.700613 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ad8d609029540708c46281d484f09fb"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "caea743c1026e75c70bcd90a5ac830a6"
server_key_iv: "7920e223ab44b96b92521f6c6e74a4be"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.701928 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:38:52.702628 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.702795 3007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:52.702891 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "9ad8d609029540708c46281d484f09fb"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "caea743c1026e75c70bcd90a5ac830a6"
server_key_iv: "7920e223ab44b96b92521f6c6e74a4be"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.702966 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:52.727381 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:52.729156 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:52.729275 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:52.729467 3007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:52.729807 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:52.729840 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.729885 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:52.729912 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.735781 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44547
I20260924 17:38:52.735824 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44547 every 8 connection(s)
I20260924 17:38:52.736133 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:52.740483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3007
I20260924 17:38:52.740585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:52.740790 392 external_mini_cluster.cc:1428] Setting key e0c05e163a0ccd765a96f32070e21a8c
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:52.745851 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.736854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39771 (local address 127.0.98.62:43937)
0924 17:38:52.736920 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:52.736922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:52.737117 (+ 195us) server_negotiation.cc:407] Connection header received
0924 17:38:52.738013 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:52.738014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:52.738025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:52.738079 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:52.738651 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.739210 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.740190 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.740469 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.740545 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.742823 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:52.742825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:52.742825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:52.742833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:52.744428 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.744798 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.744799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.744800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.744864 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.745160 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.745161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.745161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.745320 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:52.745394 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:52.745631 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 17:38:52.745719 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:38:52.745863 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.736829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43937 (local address 127.0.98.2:39771)
0924 17:38:52.736990 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:52.737027 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:52.737925 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:52.738132 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:52.738136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:52.738391 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:52.738582 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.738585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.739306 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.739307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.740111 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.740113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.740587 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.740588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.740729 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.741267 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:52.741276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:52.742740 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:52.744504 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.744509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.744511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.744720 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.744954 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.744955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.744956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.745090 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.745387 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:52.745392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:52.745501 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 17:38:52.745580 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:38:52.745707 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260924 17:38:52.746384 3136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43937
I20260924 17:38:52.746476 3136 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:52.746662 3136 heartbeater.cc:511] Master 127.0.98.62:43937 requested a full tablet report, sending...
I20260924 17:38:52.747073 2811 ts_manager.cc:194] Registered new tserver with Master: 9ad8d609029540708c46281d484f09fb (127.0.98.2:44547)
I20260924 17:38:52.747605 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39771
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:52.766582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:52.846439 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:52.846635 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:52.846660 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:52.847893 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:52.847940 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:52.847968 3141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:52.849506 3141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:52.849757 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:52.849983 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:52.854955 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.854941 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:52.854931 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:52.855268 3141 server_base.cc:1034] running on GCE node
I20260924 17:38:52.855509 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:52.855769 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:52.856931 3141 hybrid_clock.cc:648] HybridClock initialized: now 1790271532856924 us; error 34 us; skew 500 ppm
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:52.859370 3141 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:52.859730 3141 webserver.cc:460] Webserver started at http://127.0.98.3:40517/ using document root <none> and password file <none>
I20260924 17:38:52.859912 3141 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:52.859951 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:52.860035 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:52.861052 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "92a8553e83714b3eaa13d868ba08cc01"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "13ade5349303c7dcb681a5fd027112b8"
server_key_iv: "dafbe8d9c03f0eea0e31313163904b7c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.861356 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "92a8553e83714b3eaa13d868ba08cc01"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "13ade5349303c7dcb681a5fd027112b8"
server_key_iv: "dafbe8d9c03f0eea0e31313163904b7c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.862563 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:38:52.863332 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.863564 3141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:52.863665 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "92a8553e83714b3eaa13d868ba08cc01"
format_stamp: "Formatted at 2026-09-24 17:38:52 on dist-test-slave-d0pf"
server_key: "13ade5349303c7dcb681a5fd027112b8"
server_key_iv: "dafbe8d9c03f0eea0e31313163904b7c"
server_key_version: "encryptionkey@0"
I20260924 17:38:52.863719 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:52.871887 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:52.874094 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:52.874203 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:52.874377 3141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:52.874681 3141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:52.874716 3141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.874737 3141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:52.874787 3141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:52.880515 3141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44937
I20260924 17:38:52.880568 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44937 every 8 connection(s)
I20260924 17:38:52.880901 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:52.889709 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.881651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42999 (local address 127.0.98.62:43937)
0924 17:38:52.881714 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:52.881717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:52.881909 (+ 192us) server_negotiation.cc:407] Connection header received
0924 17:38:52.882838 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:52.882838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:52.882848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:52.882908 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:52.883598 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.884142 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.885167 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.885348 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.885438 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.887432 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:52.887433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:52.887434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:52.887440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:52.888600 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.888946 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.888946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.888947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.888994 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:52.889215 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:52.889216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:52.889216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:52.889308 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:52.889356 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:52.889590 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 17:38:52.889645 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:38:52.889874 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:52.881618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43937 (local address 127.0.98.3:42999)
0924 17:38:52.881780 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:52.881824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:52.882748 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:52.882967 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:52.882970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:52.883316 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:52.883522 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.883527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.884220 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.884221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.885100 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:52.885102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:52.885438 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:52.885439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:52.885538 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:52.885998 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:52.886006 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:52.887333 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:52.888666 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.888671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.888673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.888884 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.889048 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:52.889049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:52.889049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:52.889163 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:52.889369 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:52.889373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:52.889484 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 17:38:52.889565 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:38:52.889677 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260924 17:38:52.890399 3270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43937
I20260924 17:38:52.890506 3270 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:52.890677 3270 heartbeater.cc:511] Master 127.0.98.62:43937 requested a full tablet report, sending...
I20260924 17:38:52.891098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3141
I20260924 17:38:52.891140 2811 ts_manager.cc:194] Registered new tserver with Master: 92a8553e83714b3eaa13d868ba08cc01 (127.0.98.3:44937)
I20260924 17:38:52.891201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:52.891394 392 external_mini_cluster.cc:1428] Setting key 3987cf1eb929edf69cab8fd7285b3892
I20260924 17:38:52.891635 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42999
I20260924 17:38:52.892597 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 17:38:52.902978 2850 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:38:52.901970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.98.62:43937)
0924 17:38:52.902029 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:52.902031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:52.902191 (+ 160us) server_negotiation.cc:407] Connection header received
0924 17:38:52.902272 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:52.902273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:52.902290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:52.902355 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:52.902899 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53524: BlockingRecv error: recv got EOF from 127.0.0.1:53524 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:38:52.903985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2873
I20260924 17:38:52.908373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3007
I20260924 17:38:52.912694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3141
I20260924 17:38:52.916863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 2779
2026-09-24T17:38:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1794 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[3282](info): set up 2 sockets
Sep 24 17:38:52 dist-test-slave-d0pf krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260924 17:38:53.954875 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:54.075426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:54.154937 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:54.155242 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:54.155282 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:54.156472 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:54.156543 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:54.156569 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:54.156605 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:54.156634 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:54.158203 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:54.158483 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:54.158712 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:54.163517 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:54.163523 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:54.163760 3295 server_base.cc:1034] running on GCE node
W20260924 17:38:54.163523 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:54.164042 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:54.164330 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:54.165493 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790271534165470 us; error 37 us; skew 500 ppm
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:54.172452 3295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:54.172869 3295 webserver.cc:460] Webserver started at http://127.0.98.62:33343/ using document root <none> and password file <none>
I20260924 17:38:54.173050 3295 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:54.173091 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:54.173180 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:54.174193 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "ca2786fd00d8455ebce1eaf7363d1102"
format_stamp: "Formatted at 2026-09-24 17:38:54 on dist-test-slave-d0pf"
server_key: "22045c51d37ae72cb814d1e372713653"
server_key_iv: "55d1482b9e76944c8dfff194008ed9cb"
server_key_version: "encryptionkey@0"
I20260924 17:38:54.174484 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca2786fd00d8455ebce1eaf7363d1102"
format_stamp: "Formatted at 2026-09-24 17:38:54 on dist-test-slave-d0pf"
server_key: "22045c51d37ae72cb814d1e372713653"
server_key_iv: "55d1482b9e76944c8dfff194008ed9cb"
server_key_version: "encryptionkey@0"
I20260924 17:38:54.175788 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:54.176455 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:54.176640 3295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:54.176733 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "ca2786fd00d8455ebce1eaf7363d1102"
format_stamp: "Formatted at 2026-09-24 17:38:54 on dist-test-slave-d0pf"
server_key: "22045c51d37ae72cb814d1e372713653"
server_key_iv: "55d1482b9e76944c8dfff194008ed9cb"
server_key_version: "encryptionkey@0"
I20260924 17:38:54.176807 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:54.196149 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:54.198237 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:54.198397 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:54.202627 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39003
I20260924 17:38:54.202667 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39003 every 8 connection(s)
I20260924 17:38:54.203092 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:54.203967 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:54.206296 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102: Bootstrap starting.
I20260924 17:38:54.206918 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:54.207266 3363 log.cc:824] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:54.207928 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102: No bootstrap required, opened a new log
I20260924 17:38:54.209465 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260924 17:38:54.209622 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:54.209661 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2786fd00d8455ebce1eaf7363d1102, State: Initialized, Role: FOLLOWER
I20260924 17:38:54.209818 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260924 17:38:54.209896 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:54.209934 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:54.209986 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:54.210506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3295
I20260924 17:38:54.210595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:54.210768 392 external_mini_cluster.cc:1428] Setting key 082e767bf950cd06923efbc9585b1c79
I20260924 17:38:54.210843 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260924 17:38:54.210947 3363 leader_election.cc:302] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2786fd00d8455ebce1eaf7363d1102; no voters:
I20260924 17:38:54.211136 3363 leader_election.cc:288] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:54.211170 3366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:54.211282 3366 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [term 1 LEADER]: Becoming Leader. State: Replica: ca2786fd00d8455ebce1eaf7363d1102, State: Running, Role: LEADER
I20260924 17:38:54.211361 3366 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260924 17:38:54.211391 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:54.212059 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2786fd00d8455ebce1eaf7363d1102. Latest consensus state: current_term: 1 leader_uuid: "ca2786fd00d8455ebce1eaf7363d1102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } } }
I20260924 17:38:54.212056 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2786fd00d8455ebce1eaf7363d1102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2786fd00d8455ebce1eaf7363d1102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } } }
I20260924 17:38:54.212203 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:54.212217 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:54.212879 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:54.213359 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:54.214807 3377 catalog_manager.cc:1347] Generated new cluster ID: 21fbda298e064058ad99aceb1ca7b59b
I20260924 17:38:54.214856 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:54.223994 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:54.211407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.98.62:39003)
0924 17:38:54.211880 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:54.211890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:54.211923 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:38:54.212046 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:54.212053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:54.212121 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:54.212292 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:54.213371 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:54.214762 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:54.215865 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:54.216536 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:54.216642 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:54.219624 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:54.219628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:54.219629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:54.219641 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:54.222258 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:54.222676 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:54.222680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:54.222680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:54.222792 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:54.222999 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:54.223000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:54.223000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:54.223364 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:54.223536 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:54.223610 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 17:38:54.223754 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":328,"threads_started":1}
I20260924 17:38:54.224818 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:38:54.233207 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:54.233774 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:54.239185 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2786fd00d8455ebce1eaf7363d1102: Generated new TSK 0
I20260924 17:38:54.239360 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:38:54.245453 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:54.241931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.98.62:39003)
0924 17:38:54.242007 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:54.242009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:54.242124 (+ 115us) server_negotiation.cc:407] Connection header received
0924 17:38:54.242196 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:54.242197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:54.242207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:54.242278 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:54.242675 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:54.243225 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:54.243977 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:54.244204 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:54.244289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:54.244999 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:54.245000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:54.245001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:38:54.245196 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:54.245212 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:38:54.245218 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:54.245279 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:54.245342 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 17:38:54.245367 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:38:54.247970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3295
2026-09-24T17:38:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1329 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3396](info): set up 2 sockets
Sep 24 17:38:54 dist-test-slave-d0pf krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260924 17:38:55.290838 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:55.406890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:55.488945 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:55.489151 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:55.489173 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:55.490365 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:55.490408 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:55.490422 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:55.490435 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:55.490491 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:55.492012 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:55.492235 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:55.492444 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:55.497076 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:55.497238 3409 server_base.cc:1034] running on GCE node
W20260924 17:38:55.497088 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:55.497100 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:55.497561 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:55.497867 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:55.499010 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790271535498989 us; error 37 us; skew 500 ppm
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:55.501492 3409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:55.501861 3409 webserver.cc:460] Webserver started at http://127.0.98.62:36851/ using document root <none> and password file <none>
I20260924 17:38:55.502035 3409 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:55.502074 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:55.502161 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:55.503211 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "2a0139375d86dd25f7e131bac7d4b749"
server_key_iv: "6a11da1d98728afbaf1fe99e2938997d"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.503526 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "2a0139375d86dd25f7e131bac7d4b749"
server_key_iv: "6a11da1d98728afbaf1fe99e2938997d"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.504724 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:38:55.505360 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.505529 3409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:38:55.505607 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "2a0139375d86dd25f7e131bac7d4b749"
server_key_iv: "6a11da1d98728afbaf1fe99e2938997d"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.505659 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:55.517355 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:55.519608 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:55.519721 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:55.523450 3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45137
I20260924 17:38:55.523509 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45137 every 8 connection(s)
I20260924 17:38:55.523847 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:55.524832 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:55.527076 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: Bootstrap starting.
I20260924 17:38:55.527655 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:55.527951 3477 log.cc:824] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:55.528601 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: No bootstrap required, opened a new log
I20260924 17:38:55.530151 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } }
I20260924 17:38:55.530293 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:55.530342 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b44e7ec56ab4d0d9df4dd17141bff1c, State: Initialized, Role: FOLLOWER
I20260924 17:38:55.530505 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } }
I20260924 17:38:55.530567 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:55.530606 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:55.530655 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:55.531343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3409
I20260924 17:38:55.531420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:55.531586 392 external_mini_cluster.cc:1428] Setting key 002b131d77acf70fddcb1b90edfe9d63
I20260924 17:38:55.531600 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } }
I20260924 17:38:55.531741 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b44e7ec56ab4d0d9df4dd17141bff1c; no voters:
I20260924 17:38:55.531915 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:55.531962 3480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:55.532092 3480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [term 1 LEADER]: Becoming Leader. State: Replica: 7b44e7ec56ab4d0d9df4dd17141bff1c, State: Running, Role: LEADER
I20260924 17:38:55.532184 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:55.532174 3480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } }
I20260924 17:38:55.532490 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } } }
I20260924 17:38:55.532552 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:55.532648 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b44e7ec56ab4d0d9df4dd17141bff1c. Latest consensus state: current_term: 1 leader_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b44e7ec56ab4d0d9df4dd17141bff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45137 } } }
I20260924 17:38:55.532693 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c [sys.catalog]: This master's current role is: LEADER
W20260924 17:38:55.534435 3498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:38:55.534503 3498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:38:55.534557 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:55.534703 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:55.536434 3486 catalog_manager.cc:1347] Generated new cluster ID: 0dbc4681a98f4f3993b13ebe030434e6
I20260924 17:38:55.536499 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:55.544263 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:55.532092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.0.98.62:45137)
0924 17:38:55.532810 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:55.532821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:55.532858 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:38:55.532972 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:55.532977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:55.533018 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:55.533161 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:55.533904 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.535452 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.536410 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.537032 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.537147 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:55.539822 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:55.539826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:55.539827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:55.539838 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:55.542484 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.542868 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.542871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.542872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.542993 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.543227 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.543228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.543229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.543548 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:55.543712 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:55.543795 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 17:38:55.544039 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":61,"threads_started":1}
I20260924 17:38:55.544936 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:55.545400 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:55.559001 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b44e7ec56ab4d0d9df4dd17141bff1c: Generated new TSK 0
I20260924 17:38:55.559219 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:55.587815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:55.686265 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:55.686520 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:55.686565 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:55.687925 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:55.688001 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:55.688056 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:55.689642 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:55.689939 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:55.690188 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:55.695341 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:55.695353 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:55.695353 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:55.695587 3503 server_base.cc:1034] running on GCE node
I20260924 17:38:55.695806 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:55.696091 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:55.697263 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790271535697242 us; error 31 us; skew 500 ppm
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:55.699864 3503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:55.700345 3503 webserver.cc:460] Webserver started at http://127.0.98.1:40781/ using document root <none> and password file <none>
I20260924 17:38:55.700601 3503 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:55.700672 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:55.700824 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:55.701900 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc2f061781f04080bc4808e7c1a0955d"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "3065bc762b3fb09e29dc38bc587155f3"
server_key_iv: "fcc3660020141a036e345f889805df89"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.702278 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc2f061781f04080bc4808e7c1a0955d"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "3065bc762b3fb09e29dc38bc587155f3"
server_key_iv: "fcc3660020141a036e345f889805df89"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.703759 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:55.704586 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.704782 3503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:55.704892 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "dc2f061781f04080bc4808e7c1a0955d"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "3065bc762b3fb09e29dc38bc587155f3"
server_key_iv: "fcc3660020141a036e345f889805df89"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.704972 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:55.723770 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:55.725836 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:55.726030 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:55.726292 3503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:55.726615 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:55.726671 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.726714 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:55.726747 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.732748 3503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35331
I20260924 17:38:55.732831 3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35331 every 8 connection(s)
I20260924 17:38:55.733214 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:55.733490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3503
I20260924 17:38:55.733585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:55.733810 392 external_mini_cluster.cc:1428] Setting key 1a4f965c01159ab403f61296725b7fd9
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:38:55.744848 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:55.734114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39913 (local address 127.0.98.62:45137)
0924 17:38:55.734253 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:55.734255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:55.734469 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:38:55.735078 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:55.735079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:55.735094 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:55.735144 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:55.736376 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.736893 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.738558 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.738760 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.738816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:55.741583 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:55.741584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:55.741584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:55.741592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:55.743292 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.743877 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.743878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.743879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.743938 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.744180 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.744181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.744181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.744296 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:55.744348 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:55.744695 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 17:38:55.744779 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 17:38:55.745036 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:55.734117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45137 (local address 127.0.98.1:39913)
0924 17:38:55.734340 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:55.734390 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:55.734956 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:55.735222 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:55.735226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:55.735929 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:55.736258 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:55.736266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.737002 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.737003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:55.738460 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:55.738463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.738857 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.738858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:55.739069 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:55.739640 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:55.739653 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:55.741476 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:55.743381 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:55.743389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:55.743391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:55.743773 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:55.744005 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:55.744006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:55.744006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:55.744121 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:55.744353 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:55.744361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:55.744518 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 17:38:55.744673 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 17:38:55.744818 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260924 17:38:55.745736 3632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 17:38:55.745862 3632 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:55.746078 3632 heartbeater.cc:511] Master 127.0.98.62:45137 requested a full tablet report, sending...
I20260924 17:38:55.746691 3441 ts_manager.cc:194] Registered new tserver with Master: dc2f061781f04080bc4808e7c1a0955d (127.0.98.1:35331)
I20260924 17:38:55.747499 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:55.762738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:55.846616 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:55.846853 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:55.846882 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:55.848281 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:55.848348 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:55.848380 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:55.850035 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:55.850339 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:55.850603 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:55.855829 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:55.855829 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:55.856135 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:55.856112 3637 server_base.cc:1034] running on GCE node
I20260924 17:38:55.856379 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:55.856650 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:55.857828 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790271535857818 us; error 43 us; skew 500 ppm
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:55.860474 3637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:55.860896 3637 webserver.cc:460] Webserver started at http://127.0.98.2:34205/ using document root <none> and password file <none>
I20260924 17:38:55.861157 3637 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:55.861207 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:55.861361 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:55.862453 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc466b6c23fa468c9b867f17793ab7a2"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "ac3cfc73482b6128fef4aae568194958"
server_key_iv: "5d1709e3c369ba2d81c3293e4df2bc2b"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.862828 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc466b6c23fa468c9b867f17793ab7a2"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "ac3cfc73482b6128fef4aae568194958"
server_key_iv: "5d1709e3c369ba2d81c3293e4df2bc2b"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.864236 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:55.865093 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.865281 3637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:55.865391 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "cc466b6c23fa468c9b867f17793ab7a2"
format_stamp: "Formatted at 2026-09-24 17:38:55 on dist-test-slave-d0pf"
server_key: "ac3cfc73482b6128fef4aae568194958"
server_key_iv: "5d1709e3c369ba2d81c3293e4df2bc2b"
server_key_version: "encryptionkey@0"
I20260924 17:38:55.865466 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:55.876917 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:55.879081 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:55.879273 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:55.879539 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:55.879865 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:55.879921 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.879966 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:55.879999 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:55.885766 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41543
I20260924 17:38:55.885835 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41543 every 8 connection(s)
I20260924 17:38:55.886191 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:55.887735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3637
I20260924 17:38:55.887864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:55.888042 392 external_mini_cluster.cc:1428] Setting key 8616d65962014b02d4de80cf42336372
Sep 24 17:38:55 dist-test-slave-d0pf krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:38:55.896340 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:55.887018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34005 (local address 127.0.98.62:45137)
0924 17:38:55.887100 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:55.887102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:55.887280 (+ 178us) server_negotiation.cc:407] Connection header received
0924 17:38:55.887790 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:55.887792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:55.887804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:55.887862 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:55.889048 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.889547 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.890596 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.890812 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.890902 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:55.893393 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:55.893395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:55.893395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:55.893403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:55.894836 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.895336 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.895337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.895338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.895395 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:55.895650 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:55.895651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:55.895651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:55.895770 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:55.895842 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:55.896133 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 17:38:55.896241 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 17:38:55.896512 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:55.886949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45137 (local address 127.0.98.2:34005)
0924 17:38:55.887138 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:55.887181 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:55.887669 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:55.887940 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:55.887945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:55.888732 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:55.888949 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:55.888955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.889648 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.889649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:55.890497 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:55.890500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:55.890912 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:55.890913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:55.891145 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:55.891695 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:55.891706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:55.893289 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:55.894938 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:55.894945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:55.894948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:55.895245 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:55.895460 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:55.895461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:55.895461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:55.895579 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:55.895833 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:55.895839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:55.895988 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 17:38:55.896103 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:38:55.896273 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260924 17:38:55.897007 3766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45137
I20260924 17:38:55.897111 3766 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:55.897271 3766 heartbeater.cc:511] Master 127.0.98.62:45137 requested a full tablet report, sending...
I20260924 17:38:55.897632 3441 ts_manager.cc:194] Registered new tserver with Master: cc466b6c23fa468c9b867f17793ab7a2 (127.0.98.2:41543)
I20260924 17:38:55.898020 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:55.913280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:55.993633 3771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:55.993847 3771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:55.993870 3771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:55.995086 3771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:55.995124 3771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:55.995180 3771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:55.996608 3771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:55.996867 3771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:55.997117 3771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:56.002238 3777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:56.002211 3779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:56.002406 3771 server_base.cc:1034] running on GCE node
W20260924 17:38:56.002214 3776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:56.002686 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:56.002961 3771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:56.004106 3771 hybrid_clock.cc:648] HybridClock initialized: now 1790271536004104 us; error 38 us; skew 500 ppm
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:56.006733 3771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:56.007229 3771 webserver.cc:460] Webserver started at http://127.0.98.3:44221/ using document root <none> and password file <none>
I20260924 17:38:56.007431 3771 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:56.007519 3771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:56.007624 3771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:56.008688 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e709591d92b4837a616bdf8071f9bf3"
format_stamp: "Formatted at 2026-09-24 17:38:56 on dist-test-slave-d0pf"
server_key: "5194541d18c17c1e5762c6ca8fe6f273"
server_key_iv: "a3591d74d9db55a4bec8babb2922d6c4"
server_key_version: "encryptionkey@0"
I20260924 17:38:56.008993 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e709591d92b4837a616bdf8071f9bf3"
format_stamp: "Formatted at 2026-09-24 17:38:56 on dist-test-slave-d0pf"
server_key: "5194541d18c17c1e5762c6ca8fe6f273"
server_key_iv: "a3591d74d9db55a4bec8babb2922d6c4"
server_key_version: "encryptionkey@0"
I20260924 17:38:56.010313 3771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:56.010993 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:56.011161 3771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:56.011265 3771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "7e709591d92b4837a616bdf8071f9bf3"
format_stamp: "Formatted at 2026-09-24 17:38:56 on dist-test-slave-d0pf"
server_key: "5194541d18c17c1e5762c6ca8fe6f273"
server_key_iv: "a3591d74d9db55a4bec8babb2922d6c4"
server_key_version: "encryptionkey@0"
I20260924 17:38:56.011346 3771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:56.035580 3771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:56.038026 3771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:56.038185 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:56.038421 3771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:56.038730 3771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:56.038792 3771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:56.038836 3771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:56.038864 3771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:56.045329 3771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42207
I20260924 17:38:56.045382 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42207 every 8 connection(s)
I20260924 17:38:56.045742 3771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:38:56.048113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3771
I20260924 17:38:56.048214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:56.048405 392 external_mini_cluster.cc:1428] Setting key 7bbe7e3732eb56347d48ece0a5ccd859
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:56.055816 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:56.046420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32781 (local address 127.0.98.62:45137)
0924 17:38:56.046504 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:56.046506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:56.046680 (+ 174us) server_negotiation.cc:407] Connection header received
0924 17:38:56.047284 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:56.047286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:56.047298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:56.047356 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:56.048326 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.048974 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.050010 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.050209 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.050298 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:56.053009 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:56.053011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:56.053011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:56.053018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:56.054487 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:56.054904 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:56.054905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:56.054905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:56.054956 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:56.055236 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:56.055237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:56.055237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:56.055351 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:56.055401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:56.055683 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 17:38:56.055750 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:38:56.055998 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:56.046387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45137 (local address 127.0.98.3:32781)
0924 17:38:56.046556 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:56.046592 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:56.047180 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:56.047435 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:56.047439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:56.048011 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:56.048209 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:56.048213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.049090 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.049092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:56.049915 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:56.049918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.050305 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.050306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:56.050455 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:56.051121 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:56.051131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:56.052916 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:56.054594 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:56.054600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:56.054602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:56.054836 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:56.055013 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:56.055013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:56.055013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:56.055183 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:56.055411 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:56.055416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:56.055537 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 17:38:56.055641 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 17:38:56.055765 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260924 17:38:56.056466 3900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45137
I20260924 17:38:56.056567 3900 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:56.056731 3900 heartbeater.cc:511] Master 127.0.98.62:45137 requested a full tablet report, sending...
I20260924 17:38:56.057087 3441 ts_manager.cc:194] Registered new tserver with Master: 7e709591d92b4837a616bdf8071f9bf3 (127.0.98.3:42207)
I20260924 17:38:56.057435 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32781
I20260924 17:38:56.060058 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271536, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271536, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:56.083156 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:56.075821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.0.98.62:45137)
0924 17:38:56.075910 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:56.075912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:56.075926 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:38:56.075998 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:56.075998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:56.076008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:56.076083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:56.076835 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.077319 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.078057 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:56.078251 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:56.078303 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:56.080671 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:56.080673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:56.080673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:56.080680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:56.082031 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:56.082352 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:56.082353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:56.082353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:56.082404 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:56.082563 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:56.082564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:56.082564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:56.082704 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:56.082803 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:56.082957 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 17:38:56.083019 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 17:38:56.083422 3441 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35896
I20260924 17:38:56.083909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3503
I20260924 17:38:56.088339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3637
I20260924 17:38:56.092432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3771
I20260924 17:38:56.096724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3409
2026-09-24T17:38:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1850 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3912](info): set up 2 sockets
Sep 24 17:38:56 dist-test-slave-d0pf krb5kdc[3912](info): commencing operation
krb5kdc: starting...
W20260924 17:38:57.134905 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:57.252355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41533
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:57.334722 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:57.334990 3925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:57.335028 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:57.336253 3925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:57.336331 3925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:57.336357 3925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:57.336390 3925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:57.336438 3925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:57.337936 3925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:57.338210 3925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:57.338455 3925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:57.343587 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.343636 3925 server_base.cc:1034] running on GCE node
W20260924 17:38:57.343614 3933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:57.343637 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.344019 3925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:57.344343 3925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:57.345525 3925 hybrid_clock.cc:648] HybridClock initialized: now 1790271537345517 us; error 41 us; skew 500 ppm
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:57.348237 3925 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:57.348649 3925 webserver.cc:460] Webserver started at http://127.0.98.62:35587/ using document root <none> and password file <none>
I20260924 17:38:57.348856 3925 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:57.348898 3925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:57.348990 3925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:57.350028 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "4ed54f68644744b68f8dc53dceb121dc"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "97a142c8a473e733b4b8fe19cca50577"
server_key_iv: "9d92b2225eb66627a2cd092af9b241cc"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.350338 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ed54f68644744b68f8dc53dceb121dc"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "97a142c8a473e733b4b8fe19cca50577"
server_key_iv: "9d92b2225eb66627a2cd092af9b241cc"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.351720 3925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:57.352429 3940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.352633 3925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:57.352730 3925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "4ed54f68644744b68f8dc53dceb121dc"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "97a142c8a473e733b4b8fe19cca50577"
server_key_iv: "9d92b2225eb66627a2cd092af9b241cc"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.352787 3925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:57.385047 3925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:57.387118 3925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:57.387271 3925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:57.391197 3925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41533
I20260924 17:38:57.391273 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41533 every 8 connection(s)
I20260924 17:38:57.391644 3925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:57.392520 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:57.394834 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc: Bootstrap starting.
I20260924 17:38:57.395506 3993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:57.395814 3993 log.cc:824] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:57.396588 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc: No bootstrap required, opened a new log
I20260924 17:38:57.397521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 3925
I20260924 17:38:57.397629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:57.397800 392 external_mini_cluster.cc:1428] Setting key bd8b68e28e59cd199e92d433e68f2f5d
I20260924 17:38:57.398810 3993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } }
I20260924 17:38:57.398986 3993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:57.399025 3993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed54f68644744b68f8dc53dceb121dc, State: Initialized, Role: FOLLOWER
I20260924 17:38:57.399230 3993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } }
I20260924 17:38:57.399333 3993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:57.399367 3993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:57.399425 3993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:57.400627 3993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } }
I20260924 17:38:57.400723 3993 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed54f68644744b68f8dc53dceb121dc; no voters:
I20260924 17:38:57.401005 3993 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:57.401036 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:57.401140 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [term 1 LEADER]: Becoming Leader. State: Replica: 4ed54f68644744b68f8dc53dceb121dc, State: Running, Role: LEADER
I20260924 17:38:57.401223 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } }
I20260924 17:38:57.401310 3993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:57.401569 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed54f68644744b68f8dc53dceb121dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } } }
I20260924 17:38:57.401718 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:57.401917 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed54f68644744b68f8dc53dceb121dc. Latest consensus state: current_term: 1 leader_uuid: "4ed54f68644744b68f8dc53dceb121dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed54f68644744b68f8dc53dceb121dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41533 } } }
I20260924 17:38:57.402045 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:57.402405 4004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:57.402923 4004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:57.404678 4004 catalog_manager.cc:1347] Generated new cluster ID: d708c2e6a0b94806a2845c14947b0012
I20260924 17:38:57.404731 4004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:57.409041 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.398352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.98.62:41533)
0924 17:38:57.398545 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.398553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.398579 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:38:57.398784 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.398790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.398837 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.398977 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.399832 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.400826 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.401893 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.402542 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.402632 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.405513 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.405516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.405517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.405529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.407401 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.407777 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.407780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.407780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.407884 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.408091 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.408092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.408093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.408330 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.408512 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.408577 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.408714 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260924 17:38:57.430099 4004 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:57.430860 4004 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:57.435108 4004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed54f68644744b68f8dc53dceb121dc: Generated new TSK 0
I20260924 17:38:57.435307 4004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:57.461977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:57.543498 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:57.543745 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:57.543785 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:57.544997 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:57.545066 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:57.545114 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:57.546641 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:57.546912 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:57.547209 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:57.552714 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:57.552858 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:57.552757 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.552722 4019 server_base.cc:1034] running on GCE node
I20260924 17:38:57.553213 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:57.553524 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:57.554692 4019 hybrid_clock.cc:648] HybridClock initialized: now 1790271537554688 us; error 43 us; skew 500 ppm
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:57.557482 4019 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:57.557986 4019 webserver.cc:460] Webserver started at http://127.0.98.1:35111/ using document root <none> and password file <none>
I20260924 17:38:57.558228 4019 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:57.558296 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:57.558454 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:57.559597 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "0da834e815c344ba8e90e1f2e364016e"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "ac66b6e015cf2a592dcb99e0084c8ffe"
server_key_iv: "cab569038c7f339e090c65b5acb1f0cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.559965 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0da834e815c344ba8e90e1f2e364016e"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "ac66b6e015cf2a592dcb99e0084c8ffe"
server_key_iv: "cab569038c7f339e090c65b5acb1f0cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.561277 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:57.562036 4034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.562222 4019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:57.562326 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "0da834e815c344ba8e90e1f2e364016e"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "ac66b6e015cf2a592dcb99e0084c8ffe"
server_key_iv: "cab569038c7f339e090c65b5acb1f0cf"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.562400 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:57.571264 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:57.573118 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:57.573285 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:57.573515 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:57.573823 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:57.573877 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.573926 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:57.573973 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.580025 4019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40553
I20260924 17:38:57.580070 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40553 every 8 connection(s)
I20260924 17:38:57.580390 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:57.587000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4019
I20260924 17:38:57.587146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:57.587353 392 external_mini_cluster.cc:1428] Setting key 864c9cca3fe5007307e1b3ca2266a5d4
I20260924 17:38:57.590348 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.581163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45937 (local address 127.0.98.62:41533)
0924 17:38:57.581232 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.581234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.581424 (+ 190us) server_negotiation.cc:407] Connection header received
0924 17:38:57.582354 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.582355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.582365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.582432 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.583061 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.583565 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.584549 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.584757 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.584812 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.587030 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.587116 (+ 86us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.587117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.587129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.588882 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.589277 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.589279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.589279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.589351 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.589675 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.589676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.589677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.589784 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.589837 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.590201 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.590281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:38:57.590638 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.581139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41533 (local address 127.0.98.1:45937)
0924 17:38:57.581298 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:57.581341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:57.582263 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:57.582493 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:57.582496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:57.582758 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:57.582960 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.582964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.583647 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.583647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.584470 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.584472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.584833 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.584833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.584943 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.585443 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:57.585451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:57.586925 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:57.588972 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.588978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.588980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.589200 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.589445 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.589446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.589446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.589601 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.589849 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:57.589855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:57.590030 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 17:38:57.590152 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 17:38:57.590338 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260924 17:38:57.591235 4148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41533
I20260924 17:38:57.591339 4148 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:57.591544 4148 heartbeater.cc:511] Master 127.0.98.62:41533 requested a full tablet report, sending...
I20260924 17:38:57.592077 3957 ts_manager.cc:194] Registered new tserver with Master: 0da834e815c344ba8e90e1f2e364016e (127.0.98.1:40553)
I20260924 17:38:57.592705 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45937
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:57.612380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:57.693759 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:57.694027 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:57.694069 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:57.695295 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:57.695363 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:57.695407 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:57.697022 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:57.697292 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:57.697533 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:57.702566 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:57.702558 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.702626 4153 server_base.cc:1034] running on GCE node
W20260924 17:38:57.702574 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.702934 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:57.703240 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:57.704385 4153 hybrid_clock.cc:648] HybridClock initialized: now 1790271537704373 us; error 29 us; skew 500 ppm
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:57.707306 4153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:57.707768 4153 webserver.cc:460] Webserver started at http://127.0.98.2:44921/ using document root <none> and password file <none>
I20260924 17:38:57.708016 4153 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:57.708060 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:57.708205 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:57.709255 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "82beb1de49914acca2af8e680a0203c6"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "beddf137b6e466416912e2f8d0e229c4"
server_key_iv: "88b24e45cc772147603c64f1820660fb"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.709609 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "82beb1de49914acca2af8e680a0203c6"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "beddf137b6e466416912e2f8d0e229c4"
server_key_iv: "88b24e45cc772147603c64f1820660fb"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.710897 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:38:57.711676 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.711874 4153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:57.711946 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "82beb1de49914acca2af8e680a0203c6"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "beddf137b6e466416912e2f8d0e229c4"
server_key_iv: "88b24e45cc772147603c64f1820660fb"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.712055 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:57.742328 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:57.744879 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:57.745056 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:57.745313 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:57.745647 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:57.745702 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.745743 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:57.745771 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.751818 4153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42075
I20260924 17:38:57.751861 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42075 every 8 connection(s)
I20260924 17:38:57.752207 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:57.757655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4153
I20260924 17:38:57.757784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:57.757957 392 external_mini_cluster.cc:1428] Setting key 94f7db1d9cce4c6b4338c8d2fac803ee
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:57.762542 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.752960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60839 (local address 127.0.98.62:41533)
0924 17:38:57.753029 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.753031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.753228 (+ 197us) server_negotiation.cc:407] Connection header received
0924 17:38:57.754234 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.754236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.754248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.754322 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.755088 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.755681 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.756723 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.756992 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.757085 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.759508 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.759509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.759510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.759520 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.761022 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.761503 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.761504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.761504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.761553 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.761854 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.761854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.761855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.761980 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.762057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.762412 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.762479 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:38:57.762840 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.752950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41533 (local address 127.0.98.2:60839)
0924 17:38:57.753102 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:57.753137 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:57.754113 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:57.754401 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:57.754405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:57.754649 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:57.754856 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.754860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.755774 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.755775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.756629 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.756631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.757095 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.757096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.757223 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.757807 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:57.757817 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:57.759399 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:57.761104 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.761110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.761112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.761427 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.761617 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.761618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.761618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.761764 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.762064 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:57.762071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:57.762253 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 17:38:57.762377 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 17:38:57.762559 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260924 17:38:57.763391 4282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41533
I20260924 17:38:57.763499 4282 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:57.763716 4282 heartbeater.cc:511] Master 127.0.98.62:41533 requested a full tablet report, sending...
I20260924 17:38:57.764083 3957 ts_manager.cc:194] Registered new tserver with Master: 82beb1de49914acca2af8e680a0203c6 (127.0.98.2:42075)
I20260924 17:38:57.764492 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60839
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:57.782373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:57.863874 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:57.864141 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:57.864183 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:57.865491 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:57.865566 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:57.865612 4287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:57.867285 4287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:57.867571 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:57.867812 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:57.872994 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:57.872994 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.873107 4287 server_base.cc:1034] running on GCE node
W20260924 17:38:57.872996 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:57.873508 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:57.873759 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:57.874912 4287 hybrid_clock.cc:648] HybridClock initialized: now 1790271537874893 us; error 40 us; skew 500 ppm
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:57.877430 4287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:57.877908 4287 webserver.cc:460] Webserver started at http://127.0.98.3:39679/ using document root <none> and password file <none>
I20260924 17:38:57.878158 4287 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:57.878225 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:57.878376 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:57.879508 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a2aa02626514bd5aaa71a57b21b26af"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "87c18c790455ed48ed121ef804cd7b2d"
server_key_iv: "777acdcf8030ac47d6978169d49afe80"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.879865 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a2aa02626514bd5aaa71a57b21b26af"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "87c18c790455ed48ed121ef804cd7b2d"
server_key_iv: "777acdcf8030ac47d6978169d49afe80"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.881156 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:57.881840 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.882009 4287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:57.882118 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "8a2aa02626514bd5aaa71a57b21b26af"
format_stamp: "Formatted at 2026-09-24 17:38:57 on dist-test-slave-d0pf"
server_key: "87c18c790455ed48ed121ef804cd7b2d"
server_key_iv: "777acdcf8030ac47d6978169d49afe80"
server_key_version: "encryptionkey@0"
I20260924 17:38:57.882193 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:57.894434 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:57.896432 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:57.896606 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:57.896903 4287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:57.897233 4287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:57.897289 4287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.897333 4287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:57.897363 4287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:57.903857 4287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42843
I20260924 17:38:57.903903 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42843 every 8 connection(s)
I20260924 17:38:57.904258 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:38:57.907066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4287
I20260924 17:38:57.907186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:57.907411 392 external_mini_cluster.cc:1428] Setting key adeba6532e7fc762c73834d22ee75107
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:57.914256 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.905158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41569 (local address 127.0.98.62:41533)
0924 17:38:57.905225 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.905227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.905460 (+ 233us) server_negotiation.cc:407] Connection header received
0924 17:38:57.906452 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.906453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.906467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.906518 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.907417 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.908264 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.909367 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.909554 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.909612 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.911771 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.911772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.911772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.911779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.912986 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.913436 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.913437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.913438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.913514 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.913734 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.913734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.913735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.913840 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.913896 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.914129 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.914192 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:38:57.914426 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.905161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41533 (local address 127.0.98.3:41569)
0924 17:38:57.905342 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:57.905378 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:57.906324 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:57.906601 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:57.906604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:57.906922 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:57.907281 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.907288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.908370 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.908372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.909287 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:57.909290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.909629 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.909629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:57.909735 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.910212 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:57.910220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:57.911684 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:57.913066 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.913072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.913074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.913364 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.913574 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:57.913574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:57.913575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:57.913677 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:57.913894 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:57.913903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:57.914019 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 17:38:57.914102 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:38:57.914223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260924 17:38:57.914970 4416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41533
I20260924 17:38:57.915122 4416 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:57.915315 4416 heartbeater.cc:511] Master 127.0.98.62:41533 requested a full tablet report, sending...
I20260924 17:38:57.915695 3957 ts_manager.cc:194] Registered new tserver with Master: 8a2aa02626514bd5aaa71a57b21b26af (127.0.98.3:42843)
I20260924 17:38:57.916105 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41569
I20260924 17:38:57.919319 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:38:57.941505 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.934150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.0.98.1:40553)
0924 17:38:57.934326 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.934331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.934341 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:38:57.934368 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.934370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.934386 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.934453 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.934998 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.935810 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.936520 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.936829 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.936881 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.939115 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.939122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.939122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.939128 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.940480 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.940786 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.940789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.940789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.940844 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.941029 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.941030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.941030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.941157 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.941260 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.941322 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.941380 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
Sep 24 17:38:57 dist-test-slave-d0pf krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:57.949292 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:57.942746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.0.98.62:41533)
0924 17:38:57.942794 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:57.942797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:57.942806 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:57.942864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:57.942865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:57.942881 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:57.942945 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:57.943475 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.943946 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.944701 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:57.944895 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:57.944967 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:57.946982 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:57.946984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:57.946984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:57.946992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:57.948159 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.948489 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.948490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.948491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.948548 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:57.948741 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:57.948742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:57.948743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:57.948891 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:57.948999 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:57.949118 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 17:38:57.949190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:38:57.949535 3957 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48582
I20260924 17:38:57.949924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4019
I20260924 17:38:57.954824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4153
I20260924 17:38:57.959213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4287
I20260924 17:38:57.963167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 3925
2026-09-24T17:38:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1866 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:38:58 dist-test-slave-d0pf krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:38:58 dist-test-slave-d0pf krb5kdc[4428](info): set up 2 sockets
Sep 24 17:38:58 dist-test-slave-d0pf krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260924 17:38:59.002940 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:38:59.122417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:59.206847 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:59.207165 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:59.207196 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:59.208459 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:38:59.208499 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:59.208514 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:59.208527 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:38:59.208540 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:38:59.210083 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:59.210304 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:59.210494 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:59.215500 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.215500 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.215535 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.215843 4441 server_base.cc:1034] running on GCE node
I20260924 17:38:59.216184 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:59.216509 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:59.217665 4441 hybrid_clock.cc:648] HybridClock initialized: now 1790271539217649 us; error 34 us; skew 500 ppm
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:59.225672 4441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:38:59.226066 4441 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:59.226141 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:59.226301 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:59.227550 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "22c1252ab9194494ab1cf829b61709a7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "5528f19334a459a00c1549f90728e1ca"
server_key_iv: "ca5d02727bd922d064f23ea4a98ccd75"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.227924 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "22c1252ab9194494ab1cf829b61709a7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "5528f19334a459a00c1549f90728e1ca"
server_key_iv: "ca5d02727bd922d064f23ea4a98ccd75"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.229316 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:59.230100 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.230301 4441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:59.230400 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "22c1252ab9194494ab1cf829b61709a7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "5528f19334a459a00c1549f90728e1ca"
server_key_iv: "ca5d02727bd922d064f23ea4a98ccd75"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.230484 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:59.249892 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:59.252107 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:59.252236 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:59.256434 4441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45913
I20260924 17:38:59.256491 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45913 every 8 connection(s)
I20260924 17:38:59.256803 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:38:59.257375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4441
I20260924 17:38:59.257477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:38:59.257678 392 external_mini_cluster.cc:1428] Setting key 7f02dbb91e8e738a263f63d32d02cbe0
I20260924 17:38:59.257923 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:59.261351 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: Bootstrap starting.
I20260924 17:38:59.262100 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:59.262411 4508 log.cc:824] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:59.263235 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: No bootstrap required, opened a new log
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:59.265568 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260924 17:38:59.265744 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:59.265796 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c1252ab9194494ab1cf829b61709a7, State: Initialized, Role: FOLLOWER
I20260924 17:38:59.265967 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260924 17:38:59.266073 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:38:59.266108 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:38:59.266160 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:59.266959 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260924 17:38:59.267102 4508 leader_election.cc:302] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c1252ab9194494ab1cf829b61709a7; no voters:
I20260924 17:38:59.267294 4508 leader_election.cc:288] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:38:59.267359 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:59.267529 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [term 1 LEADER]: Becoming Leader. State: Replica: 22c1252ab9194494ab1cf829b61709a7, State: Running, Role: LEADER
I20260924 17:38:59.267584 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:38:59.267629 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260924 17:38:59.267947 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c1252ab9194494ab1cf829b61709a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } } }
I20260924 17:38:59.268051 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:59.268203 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c1252ab9194494ab1cf829b61709a7. Latest consensus state: current_term: 1 leader_uuid: "22c1252ab9194494ab1cf829b61709a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c1252ab9194494ab1cf829b61709a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } } }
I20260924 17:38:59.268270 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:38:59.268402 4519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:38:59.268942 4519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:38:59.269593 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.258411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.62:45913)
0924 17:38:59.258604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.258617 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.258643 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:38:59.259082 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.259091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.259164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.259287 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:59.260090 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.260975 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.262027 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.262561 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.262683 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.265462 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:59.265466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:59.265466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:59.265478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:59.267682 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.268169 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.268172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.268173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.268298 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.268524 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.268524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.268525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.268893 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:59.269060 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.269209 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.269354 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260924 17:38:59.271173 4519 catalog_manager.cc:1347] Generated new cluster ID: 77558050a24340adb2fc763bc6baccd4
I20260924 17:38:59.271231 4519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:38:59.276435 4519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:38:59.277010 4519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:38:59.289917 4519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: Generated new TSK 0
I20260924 17:38:59.290164 4519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:38:59.322088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:59.402870 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:59.403162 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:59.403193 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:59.404331 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:59.404363 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:59.404389 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:38:59.405824 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:59.406015 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:59.406210 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:59.411262 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.411296 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.411425 4534 server_base.cc:1034] running on GCE node
W20260924 17:38:59.411262 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.411794 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:59.412068 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:59.413221 4534 hybrid_clock.cc:648] HybridClock initialized: now 1790271539413207 us; error 29 us; skew 500 ppm
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:59.415781 4534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:38:59.416188 4534 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:59.416261 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:59.416411 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:59.417573 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4678a754d0c4f599232b71c3f6f8025"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "32cb9dc8207b6fb833e48c5f49575163"
server_key_iv: "8608503058b10fae0c4b3e50eea43586"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.417946 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4678a754d0c4f599232b71c3f6f8025"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "32cb9dc8207b6fb833e48c5f49575163"
server_key_iv: "8608503058b10fae0c4b3e50eea43586"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.419358 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:38:59.420087 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.420253 4534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:59.420359 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "d4678a754d0c4f599232b71c3f6f8025"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "32cb9dc8207b6fb833e48c5f49575163"
server_key_iv: "8608503058b10fae0c4b3e50eea43586"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.420435 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:59.433512 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:59.436612 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:59.436820 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:59.437117 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:59.437449 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:59.437482 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.437536 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:59.437561 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.443686 4534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36685
I20260924 17:38:59.443794 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36685 every 8 connection(s)
I20260924 17:38:59.444082 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:38:59.447180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4534
I20260924 17:38:59.447281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:38:59.447466 392 external_mini_cluster.cc:1428] Setting key 18e1b7e20a51459219cea675637d7b49
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:59.454200 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.444842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57787 (local address 127.0.98.62:45913)
0924 17:38:59.444885 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.444886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.445103 (+ 217us) server_negotiation.cc:407] Connection header received
0924 17:38:59.445598 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.445599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.445610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.445676 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:59.446677 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.447318 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.448610 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.448835 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.448915 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.451177 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:59.451179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:59.451179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:59.451187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:59.452704 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.453192 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.453194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.453194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.453262 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.453513 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.453514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.453514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.453641 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:59.453713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.454007 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.454100 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:38:59.454420 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.444818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.1:57787)
0924 17:38:59.444971 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.445010 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.445486 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.445753 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.445756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.446024 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:59.446542 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.446548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.447415 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.447419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.448500 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.448504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.448925 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.448926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.449065 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.449643 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:59.449652 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:59.451019 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:59.452835 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.452843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.452846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.453107 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.453332 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.453333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.453333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.453444 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.453715 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:59.453723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:59.453870 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.453986 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.454161 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260924 17:38:59.454933 4662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260924 17:38:59.455077 4662 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:38:59.455259 4662 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260924 17:38:59.455816 4472 ts_manager.cc:194] Registered new tserver with Master: d4678a754d0c4f599232b71c3f6f8025 (127.0.98.1:36685)
I20260924 17:38:59.456488 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:38:59.474949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:59.556147 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:59.556361 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:59.556386 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:59.557512 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:59.557551 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:59.557577 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:38:59.559060 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:59.559284 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:59.559500 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:59.564385 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.564374 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.564406 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.564702 4667 server_base.cc:1034] running on GCE node
I20260924 17:38:59.564930 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:59.565143 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:59.566298 4667 hybrid_clock.cc:648] HybridClock initialized: now 1790271539566275 us; error 36 us; skew 500 ppm
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:59.569101 4667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:38:59.569546 4667 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:59.569602 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:59.569696 4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:59.570715 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5c83332262c416a8c48afe0c8321aa2"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "4c18fa6402d071907ff1f9a5b2099fb5"
server_key_iv: "61312930971d22853b901fc1e1fb5bfa"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.571015 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5c83332262c416a8c48afe0c8321aa2"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "4c18fa6402d071907ff1f9a5b2099fb5"
server_key_iv: "61312930971d22853b901fc1e1fb5bfa"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.572399 4667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:59.573227 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.573398 4667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:38:59.573489 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "e5c83332262c416a8c48afe0c8321aa2"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "4c18fa6402d071907ff1f9a5b2099fb5"
server_key_iv: "61312930971d22853b901fc1e1fb5bfa"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.573549 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:59.587306 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:59.589315 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:59.589437 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:59.589627 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:59.589982 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:59.590018 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.590042 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:59.590055 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.595732 4667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42957
I20260924 17:38:59.595806 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42957 every 8 connection(s)
I20260924 17:38:59.596091 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:38:59.600023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4667
I20260924 17:38:59.600119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:38:59.600461 392 external_mini_cluster.cc:1428] Setting key 6632d04e28fa5bba55dbd38f9823b59f
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:59.606380 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.596923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35195 (local address 127.0.98.62:45913)
0924 17:38:59.596970 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.596972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.597218 (+ 246us) server_negotiation.cc:407] Connection header received
0924 17:38:59.597740 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.597741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.597752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.597803 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:59.598770 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.599331 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.600699 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.600935 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.601010 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.603481 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:59.603483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:59.603483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:59.603494 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:59.604932 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.605375 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.605376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.605377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.605440 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.605714 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.605715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.605716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.605837 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:59.605914 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.606204 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.606290 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:38:59.606586 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.596897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.2:35195)
0924 17:38:59.597098 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.597134 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.597635 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.597890 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.597893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.598164 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:59.598669 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.598675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.599430 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.599431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.600610 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.600613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.601024 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.601025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.601194 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.601771 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:59.601780 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:59.603376 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:59.605039 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.605047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.605050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.605295 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.605510 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.605511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.605511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.605640 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.605904 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:59.605910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:59.606058 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.606170 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.606339 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260924 17:38:59.607071 4795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260924 17:38:59.607196 4795 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:59.607372 4795 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260924 17:38:59.607791 4472 ts_manager.cc:194] Registered new tserver with Master: e5c83332262c416a8c48afe0c8321aa2 (127.0.98.2:42957)
I20260924 17:38:59.608193 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35195
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:38:59.624011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:38:59.705118 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:38:59.705333 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:38:59.705358 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:38:59.706475 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:38:59.706511 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:38:59.706537 4800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:38:59.708024 4800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:38:59.708284 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:38:59.708483 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:38:59.713382 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:38:59.713411 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.713524 4800 server_base.cc:1034] running on GCE node
W20260924 17:38:59.713382 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:38:59.713845 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:38:59.714061 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:38:59.715196 4800 hybrid_clock.cc:648] HybridClock initialized: now 1790271539715188 us; error 46 us; skew 500 ppm
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:38:59.717675 4800 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:38:59.718082 4800 fs_manager.cc:359] Metadata directory not provided
I20260924 17:38:59.718132 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:38:59.718292 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:38:59.719444 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "938c65f86a534c00b1c8f29411848ea7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "79d5a472b4997f6c0d7b5a61c419ba18"
server_key_iv: "3a9bfa3dea3d56e2d00ead930cf4f2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.719799 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "938c65f86a534c00b1c8f29411848ea7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "79d5a472b4997f6c0d7b5a61c419ba18"
server_key_iv: "3a9bfa3dea3d56e2d00ead930cf4f2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.721035 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:38:59.721709 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.721879 4800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:38:59.721972 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "938c65f86a534c00b1c8f29411848ea7"
format_stamp: "Formatted at 2026-09-24 17:38:59 on dist-test-slave-d0pf"
server_key: "79d5a472b4997f6c0d7b5a61c419ba18"
server_key_iv: "3a9bfa3dea3d56e2d00ead930cf4f2ba"
server_key_version: "encryptionkey@0"
I20260924 17:38:59.722050 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:38:59.753364 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:38:59.755318 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:38:59.755478 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:38:59.755707 4800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:38:59.756034 4800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:38:59.756086 4800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.756131 4800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:38:59.756160 4800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:38:59.761884 4800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39205
I20260924 17:38:59.761924 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39205 every 8 connection(s)
I20260924 17:38:59.762243 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:38:59 dist-test-slave-d0pf krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:38:59.769085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4800
I20260924 17:38:59.769191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:38:59.769389 392 external_mini_cluster.cc:1428] Setting key 53ff8e589eb355462751704bee339032
I20260924 17:38:59.771799 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.762948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40019 (local address 127.0.98.62:45913)
0924 17:38:59.763018 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.763020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.763227 (+ 207us) server_negotiation.cc:407] Connection header received
0924 17:38:59.763736 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.763737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.763758 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.763805 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:59.764766 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.765265 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.766240 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.766437 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.766519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.768782 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:59.768784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:59.768784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:59.768791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:59.770582 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.770943 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.770944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.770944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.770994 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.771253 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.771254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.771254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.771358 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:59.771410 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.771666 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.771731 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:38:59.771997 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.762920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.3:40019)
0924 17:38:59.763104 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.763145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.763634 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.763890 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.763899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.764156 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:38:59.764653 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.764658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.765357 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.765358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.766150 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.766152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.766528 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.766529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.766652 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.767145 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:38:59.767153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:38:59.768692 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:38:59.770656 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.770661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.770663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.770884 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.771087 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:38:59.771087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:38:59.771088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:38:59.771197 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:38:59.771415 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:38:59.771420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:38:59.771547 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.771625 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.771746 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260924 17:38:59.772447 4928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260924 17:38:59.772545 4928 heartbeater.cc:464] Registering TS with master...
I20260924 17:38:59.772701 4928 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260924 17:38:59.773051 4472 ts_manager.cc:194] Registered new tserver with Master: 938c65f86a534c00b1c8f29411848ea7 (127.0.98.3:39205)
I20260924 17:38:59.773432 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40019
I20260924 17:38:59.781229 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:38:59.788617 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.782492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.62:45913)
0924 17:38:59.782567 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.782570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.782656 (+ 86us) server_negotiation.cc:407] Connection header received
0924 17:38:59.782737 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.782737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.782758 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.782823 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:38:59.783441 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.783925 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.784636 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.784832 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.784914 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.786331 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:38:59.786332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:38:59.786333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:38:59.786341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:38:59.787499 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.787846 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.787847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.787847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.787903 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:38:59.788088 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:38:59.788089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:38:59.788090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:38:59.788233 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:38:59.788341 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.788440 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.788517 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:38:59.789988 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:38:59.791661 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:38:59.797991 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.793960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39205 (local address 127.0.0.1:43258)
0924 17:38:59.794306 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.794314 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.794361 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.794721 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.794722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.794724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:59.794772 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.794774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795655 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.795656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.796962 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.796965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797677 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.797765 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797779 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.797848 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.797907 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260924 17:38:59.798147 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.793782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36685 (local address 127.0.0.1:52636)
0924 17:38:59.794056 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.794099 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.794177 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.794507 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.794508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.794509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:59.794560 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.794562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795937 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.795938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.796860 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.796862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797883 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.797952 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797956 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.798006 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.798044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":5,"thread_start_us":48,"threads_started":1}
I20260924 17:38:59.798049 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.793782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42957 (local address 127.0.0.1:36074)
0924 17:38:59.794178 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.794188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.794242 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.794488 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.794491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.794492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:59.794560 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.794561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795914 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.795915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.796996 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.796998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797809 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.797870 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797872 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.797922 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.797952 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260924 17:38:59.798252 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.794053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.0.98.3:39205)
0924 17:38:59.794203 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.794208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.794373 (+ 165us) server_negotiation.cc:407] Connection header received
0924 17:38:59.794622 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.794624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.794643 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.794709 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:59.794828 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795563 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797101 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797591 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797679 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797744 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.798122 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.798169 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 17:38:59.798458 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.793890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.98.1:36685)
0924 17:38:59.794062 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.794067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.794268 (+ 201us) server_negotiation.cc:407] Connection header received
0924 17:38:59.794329 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.794332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.794348 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.794420 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:59.794645 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795801 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.796950 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797696 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797843 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.798298 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.798351 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260924 17:38:59.798946 4862 tablet_service.cc:1467] Processing CreateTablet for tablet c96366e959214914a9776be97a35f22d (DEFAULT_TABLE table=test-table [id=bf7b59c0ce874890a7e6425a2b42cb50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:59.799124 4596 tablet_service.cc:1467] Processing CreateTablet for tablet c96366e959214914a9776be97a35f22d (DEFAULT_TABLE table=test-table [id=bf7b59c0ce874890a7e6425a2b42cb50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:59.799319 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96366e959214914a9776be97a35f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:59.799484 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96366e959214914a9776be97a35f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:59.801811 4945 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Bootstrap starting.
I20260924 17:38:59.802115 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.793894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.0.98.2:42957)
0924 17:38:59.794136 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.794139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.794243 (+ 104us) server_negotiation.cc:407] Connection header received
0924 17:38:59.794318 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.794319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.794333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.794486 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:59.794646 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.795799 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797080 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.797749 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.797811 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.797857 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.797945 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.797995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260924 17:38:59.802357 4946 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: Bootstrap starting.
I20260924 17:38:59.802501 4945 tablet_bootstrap.cc:654] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:59.802758 4729 tablet_service.cc:1467] Processing CreateTablet for tablet c96366e959214914a9776be97a35f22d (DEFAULT_TABLE table=test-table [id=bf7b59c0ce874890a7e6425a2b42cb50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:38:59.803068 4946 tablet_bootstrap.cc:654] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:59.803093 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96366e959214914a9776be97a35f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:38:59.803165 4945 log.cc:824] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:59.803669 4946 log.cc:824] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:59.803977 4945 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: No bootstrap required, opened a new log
I20260924 17:38:59.804062 4945 ts_tablet_manager.cc:1397] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:59.804441 4946 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: No bootstrap required, opened a new log
I20260924 17:38:59.804512 4946 ts_tablet_manager.cc:1397] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:59.806048 4949 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Bootstrap starting.
I20260924 17:38:59.806069 4945 raft_consensus.cc:348] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.806250 4945 raft_consensus.cc:374] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:59.806293 4945 raft_consensus.cc:729] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938c65f86a534c00b1c8f29411848ea7, State: Initialized, Role: FOLLOWER
I20260924 17:38:59.806411 4945 consensus_queue.cc:260] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.806461 4946 raft_consensus.cc:348] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.806620 4946 raft_consensus.cc:374] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:59.806667 4946 raft_consensus.cc:729] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4678a754d0c4f599232b71c3f6f8025, State: Initialized, Role: FOLLOWER
I20260924 17:38:59.806689 4945 ts_tablet_manager.cc:1428] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:38:59.806766 4949 tablet_bootstrap.cc:654] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Neither blocks nor log segments found. Creating new log.
I20260924 17:38:59.806792 4928 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
I20260924 17:38:59.807005 4946 consensus_queue.cc:260] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.807354 4946 ts_tablet_manager.cc:1428] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:38:59.807396 4949 log.cc:824] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:38:59.807395 4662 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
I20260924 17:38:59.808208 4949 tablet_bootstrap.cc:492] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: No bootstrap required, opened a new log
I20260924 17:38:59.808292 4949 ts_tablet_manager.cc:1397] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:59.809840 4949 raft_consensus.cc:348] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.809967 4949 raft_consensus.cc:374] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:38:59.809988 4949 raft_consensus.cc:729] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c83332262c416a8c48afe0c8321aa2, State: Initialized, Role: FOLLOWER
I20260924 17:38:59.810091 4949 consensus_queue.cc:260] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.810313 4949 ts_tablet_manager.cc:1428] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:38:59.810379 4795 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
W20260924 17:38:59.846632 4796 tablet.cc:2369] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:38:59.889854 4951 raft_consensus.cc:482] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:38:59.889979 4951 raft_consensus.cc:504] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.890336 4951 leader_election.cc:288] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c83332262c416a8c48afe0c8321aa2 (127.0.98.2:42957), 938c65f86a534c00b1c8f29411848ea7 (127.0.98.3:39205)
I20260924 17:38:59.893229 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.890579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42957 (local address 127.0.98.1:53917)
0924 17:38:59.890641 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.890654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.890701 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.890911 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.890912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.890913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:59.890974 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.890975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.891640 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.891641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.892390 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.892392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.892938 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.892939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.893021 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.893024 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.893109 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.893151 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:38:59.893229 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.890695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53917 (local address 127.0.98.2:42957)
0924 17:38:59.890751 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.890753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.890762 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:38:59.890791 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.890792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.890803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.890874 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:59.891078 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.891552 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.892487 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.892854 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.892939 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.892949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.893127 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.893166 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:38:59.893369 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.890789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37685 (local address 127.0.98.3:39205)
0924 17:38:59.890863 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.890865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.890938 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:38:59.891056 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.891057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.891068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.891121 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:38:59.891273 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.891734 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.892647 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.893025 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.893110 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.893119 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.893271 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.893304 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:38:59.893384 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.890703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39205 (local address 127.0.98.1:37685)
0924 17:38:59.890884 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.890895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:38:59.890948 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:38:59.891125 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:38:59.891126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:38:59.891127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:38:59.891201 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.891202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.891821 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.891822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.892553 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:38:59.892555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.893118 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.893119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:38:59.893195 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.893199 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:38:59.893266 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:38:59.893303 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":111,"threads_started":1}
I20260924 17:38:59.893687 4749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96366e959214914a9776be97a35f22d" candidate_uuid: "d4678a754d0c4f599232b71c3f6f8025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c83332262c416a8c48afe0c8321aa2" is_pre_election: true
I20260924 17:38:59.893867 4749 raft_consensus.cc:2454] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4678a754d0c4f599232b71c3f6f8025 in term 0.
I20260924 17:38:59.894004 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96366e959214914a9776be97a35f22d" candidate_uuid: "d4678a754d0c4f599232b71c3f6f8025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938c65f86a534c00b1c8f29411848ea7" is_pre_election: true
I20260924 17:38:59.894088 4552 leader_election.cc:302] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4678a754d0c4f599232b71c3f6f8025, e5c83332262c416a8c48afe0c8321aa2; no voters:
I20260924 17:38:59.894176 4882 raft_consensus.cc:2454] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4678a754d0c4f599232b71c3f6f8025 in term 0.
I20260924 17:38:59.894232 4951 raft_consensus.cc:2781] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:38:59.894295 4951 raft_consensus.cc:482] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:38:59.894333 4951 raft_consensus.cc:3037] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:59.895004 4951 raft_consensus.cc:504] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.895179 4951 leader_election.cc:288] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [CANDIDATE]: Term 1 election: Requested vote from peers e5c83332262c416a8c48afe0c8321aa2 (127.0.98.2:42957), 938c65f86a534c00b1c8f29411848ea7 (127.0.98.3:39205)
I20260924 17:38:59.895381 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96366e959214914a9776be97a35f22d" candidate_uuid: "d4678a754d0c4f599232b71c3f6f8025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938c65f86a534c00b1c8f29411848ea7"
I20260924 17:38:59.895397 4749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96366e959214914a9776be97a35f22d" candidate_uuid: "d4678a754d0c4f599232b71c3f6f8025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c83332262c416a8c48afe0c8321aa2"
I20260924 17:38:59.895474 4749 raft_consensus.cc:3037] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:59.895474 4882 raft_consensus.cc:3037] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:38:59.896346 4749 raft_consensus.cc:2454] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4678a754d0c4f599232b71c3f6f8025 in term 1.
I20260924 17:38:59.896346 4882 raft_consensus.cc:2454] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4678a754d0c4f599232b71c3f6f8025 in term 1.
I20260924 17:38:59.896520 4552 leader_election.cc:302] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 938c65f86a534c00b1c8f29411848ea7, d4678a754d0c4f599232b71c3f6f8025; no voters:
I20260924 17:38:59.896638 4951 raft_consensus.cc:2781] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:38:59.896814 4951 raft_consensus.cc:686] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 1 LEADER]: Becoming Leader. State: Replica: d4678a754d0c4f599232b71c3f6f8025, State: Running, Role: LEADER
I20260924 17:38:59.896924 4951 consensus_queue.cc:237] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260924 17:38:59.897782 4471 catalog_manager.cc:5495] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4678a754d0c4f599232b71c3f6f8025 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d4678a754d0c4f599232b71c3f6f8025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4678a754d0c4f599232b71c3f6f8025" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260924 17:38:59.913043 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.910935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.98.1:36685)
0924 17:38:59.910994 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:38:59.911000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:38:59.911008 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:38:59.911075 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:38:59.911076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:38:59.911103 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:38:59.911163 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:38:59.911288 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.911731 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.912419 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:38:59.912592 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:38:59.912641 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:38:59.912774 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:38:59.912874 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:38:59.912935 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:38:59.912954 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 17:38:59.912970 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:38:59.914757 4882 raft_consensus.cc:1261] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 1 FOLLOWER]: Refusing update from remote peer d4678a754d0c4f599232b71c3f6f8025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:59.914757 4749 raft_consensus.cc:1261] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 1 FOLLOWER]: Refusing update from remote peer d4678a754d0c4f599232b71c3f6f8025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:38:59.915104 4951 consensus_queue.cc:1035] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938c65f86a534c00b1c8f29411848ea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:59.915244 4955 consensus_queue.cc:1035] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c83332262c416a8c48afe0c8321aa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:38:59.917546 4958 mvcc.cc:205] Tried to move back new op lower bound from 7332952227487793152 to 7332952227418267648. Current Snapshot: MvccSnapshot[applied={T|T < 7332952227487793152}]
I20260924 17:38:59.918473 4960 mvcc.cc:205] Tried to move back new op lower bound from 7332952227487793152 to 7332952227418267648. Current Snapshot: MvccSnapshot[applied={T|T < 7332952227487793152}]
I20260924 17:38:59.919785 4472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:59.919862 4472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:38:59.920614 4472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22c1252ab9194494ab1cf829b61709a7: Sending DeleteTablet for 3 replicas of tablet c96366e959214914a9776be97a35f22d
I20260924 17:38:59.920930 4596 tablet_service.cc:1514] Processing DeleteTablet for tablet c96366e959214914a9776be97a35f22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52636
I20260924 17:38:59.920931 4729 tablet_service.cc:1514] Processing DeleteTablet for tablet c96366e959214914a9776be97a35f22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36074
I20260924 17:38:59.921132 4968 tablet_replica.cc:330] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025: stopping tablet replica
I20260924 17:38:59.921164 4862 tablet_service.cc:1514] Processing DeleteTablet for tablet c96366e959214914a9776be97a35f22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43258
I20260924 17:38:59.921231 4968 raft_consensus.cc:2229] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:38:59.921325 4968 raft_consensus.cc:2258] T c96366e959214914a9776be97a35f22d P d4678a754d0c4f599232b71c3f6f8025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:59.921586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4534
I20260924 17:38:59.921522 4970 tablet_replica.cc:330] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: stopping tablet replica
I20260924 17:38:59.922331 4970 raft_consensus.cc:2229] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:59.922425 4970 raft_consensus.cc:2258] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:59.921368 4969 tablet_replica.cc:330] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: stopping tablet replica
I20260924 17:38:59.922576 4969 raft_consensus.cc:2229] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:38:59.922677 4969 raft_consensus.cc:2258] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:38:59.923147 4969 ts_tablet_manager.cc:1905] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:59.923148 4970 ts_tablet_manager.cc:1905] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:38:59.924932 4969 ts_tablet_manager.cc:1918] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:59.924932 4970 ts_tablet_manager.cc:1918] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:38:59.925012 4970 log.cc:1196] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/c96366e959214914a9776be97a35f22d
I20260924 17:38:59.925012 4969 log.cc:1196] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/c96366e959214914a9776be97a35f22d
I20260924 17:38:59.925311 4969 ts_tablet_manager.cc:1939] T c96366e959214914a9776be97a35f22d P e5c83332262c416a8c48afe0c8321aa2: Deleting consensus metadata
I20260924 17:38:59.925311 4970 ts_tablet_manager.cc:1939] T c96366e959214914a9776be97a35f22d P 938c65f86a534c00b1c8f29411848ea7: Deleting consensus metadata
I20260924 17:38:59.925714 4459 catalog_manager.cc:4841] TS e5c83332262c416a8c48afe0c8321aa2 (127.0.98.2:42957): tablet c96366e959214914a9776be97a35f22d (table test-table [id=bf7b59c0ce874890a7e6425a2b42cb50]) successfully deleted
I20260924 17:38:59.925836 4459 catalog_manager.cc:4841] TS 938c65f86a534c00b1c8f29411848ea7 (127.0.98.3:39205): tablet c96366e959214914a9776be97a35f22d (table test-table [id=bf7b59c0ce874890a7e6425a2b42cb50]) successfully deleted
W20260924 17:38:59.927220 4456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36685 (error 108)
I20260924 17:38:59.927762 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:38:59.927625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36685 (local address 127.0.0.1:52664)
0924 17:38:59.927676 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:38:59.927708 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:38:59.927801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4667
W20260924 17:38:59.927901 4456 catalog_manager.cc:4584] TS d4678a754d0c4f599232b71c3f6f8025 (127.0.98.1:36685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c96366e959214914a9776be97a35f22d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36685: connect: Connection refused (error 111)
I20260924 17:38:59.932224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4800
I20260924 17:38:59.936827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4441
2026-09-24T17:38:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1973 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T17:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:38:59Z Disabled control of system clock
I20260924 17:38:59.963819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34105
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 17:39:00.045778 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.046018 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.046041 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.047335 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:00.047384 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.047398 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.047411 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:00.047423 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:00.048892 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.049185 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.049512 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:00.054682 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.054735 4975 server_base.cc:1034] running on GCE node
W20260924 17:39:00.054672 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.054668 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.055178 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:00.055492 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:00.056664 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790271540056652 us; error 40 us; skew 500 ppm
I20260924 17:39:00.058105 4975 webserver.cc:460] Webserver started at http://127.0.98.62:34435/ using document root <none> and password file <none>
I20260924 17:39:00.058378 4975 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:00.058450 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:00.058597 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:00.059749 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "0952dd451a4f4c01b50e8a9198654091"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "2c95fa3b567cbfd9545ab71633e684bb"
server_key_iv: "849a0e67933763ef50d59ca2fb6a1c75"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.060114 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "0952dd451a4f4c01b50e8a9198654091"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "2c95fa3b567cbfd9545ab71633e684bb"
server_key_iv: "849a0e67933763ef50d59ca2fb6a1c75"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.061444 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:00.062216 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.062425 4975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:00.062549 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0952dd451a4f4c01b50e8a9198654091"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "2c95fa3b567cbfd9545ab71633e684bb"
server_key_iv: "849a0e67933763ef50d59ca2fb6a1c75"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.062634 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:00.066111 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:00.066411 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:00.066556 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:00.070580 4975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34105
I20260924 17:39:00.070648 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34105 every 8 connection(s)
I20260924 17:39:00.071009 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:00.071724 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:00.074460 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: Bootstrap starting.
I20260924 17:39:00.075143 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:00.075453 5042 log.cc:824] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:00.076148 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: No bootstrap required, opened a new log
I20260924 17:39:00.077749 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 17:39:00.077908 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:00.077932 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0952dd451a4f4c01b50e8a9198654091, State: Initialized, Role: FOLLOWER
I20260924 17:39:00.078130 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 17:39:00.078186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 4975
I20260924 17:39:00.078233 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:00.078270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:00.078271 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:00.078315 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:00.078466 392 external_mini_cluster.cc:1428] Setting key 06bfd0117c5695f37e709d3c19ccae91
I20260924 17:39:00.079022 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 17:39:00.079169 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0952dd451a4f4c01b50e8a9198654091; no voters:
I20260924 17:39:00.079331 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:00.079387 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:00.079562 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [term 1 LEADER]: Becoming Leader. State: Replica: 0952dd451a4f4c01b50e8a9198654091, State: Running, Role: LEADER
I20260924 17:39:00.079703 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 17:39:00.079581 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:00.080338 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0952dd451a4f4c01b50e8a9198654091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } } }
I20260924 17:39:00.080417 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:00.080833 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0952dd451a4f4c01b50e8a9198654091. Latest consensus state: current_term: 1 leader_uuid: "0952dd451a4f4c01b50e8a9198654091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0952dd451a4f4c01b50e8a9198654091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } } }
I20260924 17:39:00.080929 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:00.081022 5051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:00.081079 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.078986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.0.98.62:34105)
0924 17:39:00.079249 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.079260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.079298 (+ 38us) server_negotiation.cc:407] Connection header received
0924 17:39:00.079441 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.079449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.079496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.079688 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.080406 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.080409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.080410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.080413 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.080456 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.080472 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.080778 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.080815 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.080884 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20260924 17:39:00.081661 5051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:00.083293 5051 catalog_manager.cc:1347] Generated new cluster ID: f26440dc480b4cf9a6a6b3a8d46594f0
I20260924 17:39:00.083355 5051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:00.096405 5051 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:00.096603 5051 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:00.102372 5051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: Generated new TSK 0
I20260924 17:39:00.102588 5051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:00.108470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--builtin_ntp_servers=127.0.98.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 17:39:00.191066 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.191347 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.191391 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.192648 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.192725 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.192770 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:00.194195 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.194509 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.194830 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:00.201251 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.201421 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.201251 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.201309 5066 server_base.cc:1034] running on GCE node
I20260924 17:39:00.201794 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:00.202087 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:00.203267 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790271540203245 us; error 33 us; skew 500 ppm
I20260924 17:39:00.205051 5066 webserver.cc:460] Webserver started at http://127.0.98.1:45125/ using document root <none> and password file <none>
I20260924 17:39:00.205399 5066 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:00.205470 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:00.205618 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:00.207075 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "32aba1d919384e92b2911f228fc14a17"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "ccbc1bb19c3930ffb677f1547ddbcaf0"
server_key_iv: "cdd1e41c1d2423061f654a0b70f80137"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.207508 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32aba1d919384e92b2911f228fc14a17"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "ccbc1bb19c3930ffb677f1547ddbcaf0"
server_key_iv: "cdd1e41c1d2423061f654a0b70f80137"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.209204 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:00.210181 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.210526 5066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:00.210623 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "32aba1d919384e92b2911f228fc14a17"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "ccbc1bb19c3930ffb677f1547ddbcaf0"
server_key_iv: "cdd1e41c1d2423061f654a0b70f80137"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.210701 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:00.215669 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:00.215924 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:00.216022 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:00.216179 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:00.216511 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:00.216547 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.216570 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:00.216622 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.222645 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38461
I20260924 17:39:00.222680 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38461 every 8 connection(s)
I20260924 17:39:00.223080 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:39:00.223601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5066
I20260924 17:39:00.223731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:00.223961 392 external_mini_cluster.cc:1428] Setting key e696319bb6131ad59c5ddb7e57f1e0da
I20260924 17:39:00.224985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--builtin_ntp_servers=127.0.98.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 17:39:00.226444 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.223898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56093 (local address 127.0.98.62:34105)
0924 17:39:00.223973 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.223975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.224264 (+ 289us) server_negotiation.cc:407] Connection header received
0924 17:39:00.224836 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.224837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.224843 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.224917 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.226198 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.226199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.226200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.226204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.226220 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.226224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.226276 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.226378 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.226391 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:39:00.226712 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.223918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34105 (local address 127.0.98.1:56093)
0924 17:39:00.224121 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.224164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.224744 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.224979 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.224983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.224991 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.225984 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.225984 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.226031 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.226077 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.226292 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.226297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.226301 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.226392 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.226467 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260924 17:39:00.227350 5194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34105
I20260924 17:39:00.227454 5194 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:00.227608 5194 heartbeater.cc:511] Master 127.0.98.62:34105 requested a full tablet report, sending...
I20260924 17:39:00.228506 5006 ts_manager.cc:194] Registered new tserver with Master: 32aba1d919384e92b2911f228fc14a17 (127.0.98.1:38461)
W20260924 17:39:00.312434 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.312671 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.312697 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.313966 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.314011 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.314040 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:00.315655 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.315948 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.316305 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:00.322103 5197 server_base.cc:1034] running on GCE node
W20260924 17:39:00.322124 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.322084 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.322073 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.322525 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:00.322794 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:00.323961 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790271540323945 us; error 28 us; skew 500 ppm
I20260924 17:39:00.325177 5197 webserver.cc:460] Webserver started at http://127.0.98.2:33765/ using document root <none> and password file <none>
I20260924 17:39:00.325431 5197 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:00.325502 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:00.325656 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:00.326727 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "454deb72c23f45b08427c975afe228e8"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "086967b4210dae9968077c421e0a8b35"
server_key_iv: "93829eac84162315af93161403b09293"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.327158 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "454deb72c23f45b08427c975afe228e8"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "086967b4210dae9968077c421e0a8b35"
server_key_iv: "93829eac84162315af93161403b09293"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.328545 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:00.329334 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.329634 5197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:00.329734 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "454deb72c23f45b08427c975afe228e8"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "086967b4210dae9968077c421e0a8b35"
server_key_iv: "93829eac84162315af93161403b09293"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.329826 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:00.333640 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:00.333935 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:00.334084 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:00.334311 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:00.334626 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:00.334687 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.334731 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:00.334758 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.340706 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37551
I20260924 17:39:00.340780 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37551 every 8 connection(s)
I20260924 17:39:00.341138 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:00.344502 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.341916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41531 (local address 127.0.98.62:34105)
0924 17:39:00.341990 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.341993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.342387 (+ 394us) server_negotiation.cc:407] Connection header received
0924 17:39:00.342813 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.342815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.342821 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.342902 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.344278 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.344279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.344280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.344284 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.344300 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.344304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.344369 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.344432 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.344445 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:39:00.344697 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.341916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34105 (local address 127.0.98.2:41531)
0924 17:39:00.342197 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.342236 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.342709 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.342981 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.342985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.342993 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.344096 (+ 1103us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.344097 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.344129 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.344163 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.344366 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.344368 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.344371 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.344426 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.344479 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
I20260924 17:39:00.345287 5325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34105
I20260924 17:39:00.345407 5325 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:00.345567 5325 heartbeater.cc:511] Master 127.0.98.62:34105 requested a full tablet report, sending...
I20260924 17:39:00.345947 5006 ts_manager.cc:194] Registered new tserver with Master: 454deb72c23f45b08427c975afe228e8 (127.0.98.2:37551)
I20260924 17:39:00.350662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5197
I20260924 17:39:00.350800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:00.350988 392 external_mini_cluster.cc:1428] Setting key 22434d9e0b2784b3422d56683420a11f
I20260924 17:39:00.352027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--builtin_ntp_servers=127.0.98.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 17:39:00.435914 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.436184 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.436225 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.437754 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.437815 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.437860 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:00.439996 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.440311 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.440670 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:00.446066 5328 server_base.cc:1034] running on GCE node
W20260924 17:39:00.446053 5336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.446048 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.446033 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.446527 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:00.446825 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:00.447991 5328 hybrid_clock.cc:648] HybridClock initialized: now 1790271540447960 us; error 53 us; skew 500 ppm
I20260924 17:39:00.449275 5328 webserver.cc:460] Webserver started at http://127.0.98.3:38553/ using document root <none> and password file <none>
I20260924 17:39:00.449533 5328 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:00.449604 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:00.449739 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:00.450817 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "299c54fabdc24948999d1c7eaabf6c8b"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "97b72ad6a94faaaa22dece0278401463"
server_key_iv: "ada3bb5c6d8b40f25bc20a4c7be94afe"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.451220 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "299c54fabdc24948999d1c7eaabf6c8b"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "97b72ad6a94faaaa22dece0278401463"
server_key_iv: "ada3bb5c6d8b40f25bc20a4c7be94afe"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.452584 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:00.453403 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.453594 5328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:00.453691 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "299c54fabdc24948999d1c7eaabf6c8b"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "97b72ad6a94faaaa22dece0278401463"
server_key_iv: "ada3bb5c6d8b40f25bc20a4c7be94afe"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.453775 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:00.457316 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:00.457595 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:00.457733 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:00.457950 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:00.458249 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:00.458302 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.458339 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:00.458369 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.463940 5328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42293
I20260924 17:39:00.464017 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42293 every 8 connection(s)
I20260924 17:39:00.464329 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:00.466886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5328
I20260924 17:39:00.467010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:00.467249 392 external_mini_cluster.cc:1428] Setting key bd9d00fc8365808008f4e428526a3e49
I20260924 17:39:00.467561 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.465017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55743 (local address 127.0.98.62:34105)
0924 17:39:00.465084 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.465087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.465388 (+ 301us) server_negotiation.cc:407] Connection header received
0924 17:39:00.465840 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.465841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.465846 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.465896 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.467276 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.467277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.467277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.467281 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.467310 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.467316 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.467397 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.467476 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.467491 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:39:00.467864 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.465008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34105 (local address 127.0.98.3:55743)
0924 17:39:00.465222 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.465275 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.465759 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.465948 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.465951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.465959 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.467100 (+ 1141us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.467101 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.467143 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.467183 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.467395 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.467402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.467405 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.467488 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.467548 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260924 17:39:00.468348 5456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34105
I20260924 17:39:00.468430 5456 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:00.468614 5456 heartbeater.cc:511] Master 127.0.98.62:34105 requested a full tablet report, sending...
I20260924 17:39:00.468950 5006 ts_manager.cc:194] Registered new tserver with Master: 299c54fabdc24948999d1c7eaabf6c8b (127.0.98.3:42293)
I20260924 17:39:00.479126 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:39:00.481743 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.480516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.0.98.62:34105)
0924 17:39:00.480555 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.480558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.480682 (+ 124us) server_negotiation.cc:407] Connection header received
0924 17:39:00.480753 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.480754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.480760 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.480815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.481515 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.481516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.481517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.481521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.481535 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.481540 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.481594 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.481656 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.481672 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:39:00.483422 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:00.484869 5006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:00.488456 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.486921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37551 (local address 127.0.0.1:35012)
0924 17:39:00.487110 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.487143 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.487197 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.487498 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.487502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.487507 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.488058 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.488058 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.488063 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.488073 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.488286 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.488292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.488296 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.488367 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.488382 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260924 17:39:00.488469 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.487162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.98.2:37551)
0924 17:39:00.487353 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.487359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.487369 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:00.487397 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.487399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.487410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.487490 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.488132 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.488139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.488139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.488143 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.488182 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.488197 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.488292 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.488376 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.488391 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
I20260924 17:39:00.489215 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.486915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42293 (local address 127.0.0.1:55778)
0924 17:39:00.487099 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.487142 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.487197 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.487572 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.487573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.487575 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.487958 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.487959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.487972 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.488014 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.489078 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.489079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.489080 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.489136 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.489150 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20260924 17:39:00.489238 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 (DEFAULT_TABLE table=test-table [id=1b20ebf035fe4631964504a7714b9faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:00.489418 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.487084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.3:42293)
0924 17:39:00.487442 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.487446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.487455 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:00.487480 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.487482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.487489 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.487559 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.488904 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.488908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.488908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.488912 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.488937 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.488950 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.489058 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.489319 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.489335 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":78,"threads_started":1}
I20260924 17:39:00.489563 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.486921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38461 (local address 127.0.0.1:52746)
0924 17:39:00.487284 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.487292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.487324 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.488603 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.488604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.488608 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.489032 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.489032 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.489038 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.489052 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.489243 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.489243 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.489244 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.489466 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.489480 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 17:39:00.489603 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03620734eadc4e899f40b41d7d6f9fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:00.490072 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 (DEFAULT_TABLE table=test-table [id=1b20ebf035fe4631964504a7714b9faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:00.490396 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03620734eadc4e899f40b41d7d6f9fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:00.491389 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.487251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.0.98.1:38461)
0924 17:39:00.488196 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.488201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.488210 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:00.488235 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.488237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.488247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.488317 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.489109 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.489116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.489117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.489121 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.489150 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.489172 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.489248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.491288 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.491308 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":68,"threads_started":1}
I20260924 17:39:00.492393 5470 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Bootstrap starting.
I20260924 17:39:00.492887 5471 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Bootstrap starting.
I20260924 17:39:00.493062 5470 tablet_bootstrap.cc:654] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:00.493367 5470 log.cc:824] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:00.493498 5471 tablet_bootstrap.cc:654] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:00.493482 5128 tablet_service.cc:1467] Processing CreateTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 (DEFAULT_TABLE table=test-table [id=1b20ebf035fe4631964504a7714b9faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:00.493749 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03620734eadc4e899f40b41d7d6f9fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:00.493798 5471 log.cc:824] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:00.494130 5470 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: No bootstrap required, opened a new log
I20260924 17:39:00.494186 5470 ts_tablet_manager.cc:1397] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:00.494410 5471 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: No bootstrap required, opened a new log
I20260924 17:39:00.494449 5471 ts_tablet_manager.cc:1397] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:00.496496 5471 raft_consensus.cc:348] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.496644 5471 raft_consensus.cc:374] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:00.496686 5471 raft_consensus.cc:729] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299c54fabdc24948999d1c7eaabf6c8b, State: Initialized, Role: FOLLOWER
I20260924 17:39:00.496737 5470 raft_consensus.cc:348] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.496809 5471 consensus_queue.cc:260] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.496893 5470 raft_consensus.cc:374] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:00.496932 5470 raft_consensus.cc:729] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454deb72c23f45b08427c975afe228e8, State: Initialized, Role: FOLLOWER
I20260924 17:39:00.496974 5474 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: Bootstrap starting.
I20260924 17:39:00.497042 5470 consensus_queue.cc:260] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.497146 5471 ts_tablet_manager.cc:1428] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:39:00.497263 5456 heartbeater.cc:503] Master 127.0.98.62:34105 was elected leader, sending a full tablet report...
I20260924 17:39:00.497298 5470 ts_tablet_manager.cc:1428] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:39:00.497419 5325 heartbeater.cc:503] Master 127.0.98.62:34105 was elected leader, sending a full tablet report...
I20260924 17:39:00.497658 5474 tablet_bootstrap.cc:654] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:00.498000 5474 log.cc:824] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:00.498744 5474 tablet_bootstrap.cc:492] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: No bootstrap required, opened a new log
I20260924 17:39:00.498809 5474 ts_tablet_manager.cc:1397] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:00.500594 5474 raft_consensus.cc:348] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.500713 5474 raft_consensus.cc:374] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:00.500735 5474 raft_consensus.cc:729] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aba1d919384e92b2911f228fc14a17, State: Initialized, Role: FOLLOWER
I20260924 17:39:00.500808 5474 consensus_queue.cc:260] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.500984 5474 ts_tablet_manager.cc:1428] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:39:00.501053 5194 heartbeater.cc:503] Master 127.0.98.62:34105 was elected leader, sending a full tablet report...
W20260924 17:39:00.591696 5326 tablet.cc:2369] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:00.603503 5478 raft_consensus.cc:482] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:00.603657 5478 raft_consensus.cc:504] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.604034 5478 leader_election.cc:288] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 299c54fabdc24948999d1c7eaabf6c8b (127.0.98.3:42293), 454deb72c23f45b08427c975afe228e8 (127.0.98.2:37551)
I20260924 17:39:00.605441 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.604210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37551 (local address 127.0.98.1:44783)
0924 17:39:00.604261 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.604272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.604308 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.604551 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.604552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.604554 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.605116 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.605116 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.605121 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.605130 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.605297 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.605298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.605299 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.605349 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.605363 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:39:00.605456 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.604295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44783 (local address 127.0.98.2:37551)
0924 17:39:00.604368 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.604370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.604389 (+ 19us) server_negotiation.cc:407] Connection header received
0924 17:39:00.604420 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.604421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.604445 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.604513 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.605214 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.605215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.605216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.605219 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.605234 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.605239 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.605294 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.605358 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.605387 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:39:00.605456 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.604273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52121 (local address 127.0.98.3:42293)
0924 17:39:00.604323 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.604325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.604451 (+ 126us) server_negotiation.cc:407] Connection header received
0924 17:39:00.604477 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.604477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.604482 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.604538 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.605201 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.605202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.605203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.605208 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.605226 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.605232 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.605294 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.605369 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.605382 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:39:00.605453 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.604220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42293 (local address 127.0.98.1:52121)
0924 17:39:00.604382 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.604395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.604432 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.604551 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.604552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.604554 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:00.605084 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:00.605085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:00.605094 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:00.605109 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:00.605297 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:00.605298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:00.605299 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:00.605367 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 17:39:00.605377 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260924 17:39:00.605784 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03620734eadc4e899f40b41d7d6f9fe2" candidate_uuid: "32aba1d919384e92b2911f228fc14a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "454deb72c23f45b08427c975afe228e8" is_pre_election: true
I20260924 17:39:00.605922 5279 raft_consensus.cc:2454] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aba1d919384e92b2911f228fc14a17 in term 0.
I20260924 17:39:00.606145 5084 leader_election.cc:302] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aba1d919384e92b2911f228fc14a17, 454deb72c23f45b08427c975afe228e8; no voters:
I20260924 17:39:00.606135 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03620734eadc4e899f40b41d7d6f9fe2" candidate_uuid: "32aba1d919384e92b2911f228fc14a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299c54fabdc24948999d1c7eaabf6c8b" is_pre_election: true
I20260924 17:39:00.606271 5410 raft_consensus.cc:2454] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aba1d919384e92b2911f228fc14a17 in term 0.
I20260924 17:39:00.606267 5478 raft_consensus.cc:2781] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:00.606313 5478 raft_consensus.cc:482] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:00.606336 5478 raft_consensus.cc:3037] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:00.607173 5478 raft_consensus.cc:504] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.607312 5478 leader_election.cc:288] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [CANDIDATE]: Term 1 election: Requested vote from peers 299c54fabdc24948999d1c7eaabf6c8b (127.0.98.3:42293), 454deb72c23f45b08427c975afe228e8 (127.0.98.2:37551)
I20260924 17:39:00.607470 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03620734eadc4e899f40b41d7d6f9fe2" candidate_uuid: "32aba1d919384e92b2911f228fc14a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299c54fabdc24948999d1c7eaabf6c8b"
I20260924 17:39:00.607491 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03620734eadc4e899f40b41d7d6f9fe2" candidate_uuid: "32aba1d919384e92b2911f228fc14a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "454deb72c23f45b08427c975afe228e8"
I20260924 17:39:00.607553 5410 raft_consensus.cc:3037] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:00.607553 5279 raft_consensus.cc:3037] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:00.608422 5279 raft_consensus.cc:2454] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aba1d919384e92b2911f228fc14a17 in term 1.
I20260924 17:39:00.608422 5410 raft_consensus.cc:2454] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aba1d919384e92b2911f228fc14a17 in term 1.
I20260924 17:39:00.608610 5084 leader_election.cc:302] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aba1d919384e92b2911f228fc14a17, 454deb72c23f45b08427c975afe228e8; no voters:
I20260924 17:39:00.608723 5478 raft_consensus.cc:2781] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:00.608934 5478 raft_consensus.cc:686] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 1 LEADER]: Becoming Leader. State: Replica: 32aba1d919384e92b2911f228fc14a17, State: Running, Role: LEADER
I20260924 17:39:00.609043 5478 consensus_queue.cc:237] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } }
I20260924 17:39:00.609905 5005 catalog_manager.cc:5495] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32aba1d919384e92b2911f228fc14a17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "32aba1d919384e92b2911f228fc14a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aba1d919384e92b2911f228fc14a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 } health_report { overall_health: UNKNOWN } } }
I20260924 17:39:00.634946 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.633839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.0.98.1:38461)
0924 17:39:00.633908 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.633910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.633921 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:00.634004 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.634005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.634020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.634092 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.634726 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.634728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.634728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.634733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.634750 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.634754 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.634806 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.634848 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.634863 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:39:00.636795 5410 raft_consensus.cc:1261] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 1 FOLLOWER]: Refusing update from remote peer 32aba1d919384e92b2911f228fc14a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:00.636844 5279 raft_consensus.cc:1261] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 1 FOLLOWER]: Refusing update from remote peer 32aba1d919384e92b2911f228fc14a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:00.637197 5480 consensus_queue.cc:1035] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "454deb72c23f45b08427c975afe228e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:00.637336 5478 consensus_queue.cc:1035] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299c54fabdc24948999d1c7eaabf6c8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:00.639608 5483 mvcc.cc:205] Tried to move back new op lower bound from 7332952230445273088 to 7332952230335254528. Current Snapshot: MvccSnapshot[applied={T|T < 7332952230445273088}]
I20260924 17:39:00.639851 5484 mvcc.cc:205] Tried to move back new op lower bound from 7332952230445273088 to 7332952230335254528. Current Snapshot: MvccSnapshot[applied={T|T < 7332952230445273088}]
I20260924 17:39:00.641921 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:00.642025 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:00.642812 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0952dd451a4f4c01b50e8a9198654091: Sending DeleteTablet for 3 replicas of tablet 03620734eadc4e899f40b41d7d6f9fe2
I20260924 17:39:00.643158 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:00 UTC) from {username='slave'} at 127.0.0.1:52746
I20260924 17:39:00.643231 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:00 UTC) from {username='slave'} at 127.0.0.1:35012
I20260924 17:39:00.643249 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 03620734eadc4e899f40b41d7d6f9fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:00 UTC) from {username='slave'} at 127.0.0.1:55778
I20260924 17:39:00.643441 5494 tablet_replica.cc:330] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: stopping tablet replica
I20260924 17:39:00.643456 5495 tablet_replica.cc:330] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: stopping tablet replica
I20260924 17:39:00.643579 5493 tablet_replica.cc:330] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17: stopping tablet replica
I20260924 17:39:00.643674 5493 raft_consensus.cc:2229] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:00.643676 5495 raft_consensus.cc:2229] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:00.643751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5066
I20260924 17:39:00.643774 5493 raft_consensus.cc:2258] T 03620734eadc4e899f40b41d7d6f9fe2 P 32aba1d919384e92b2911f228fc14a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:00.643895 5495 raft_consensus.cc:2258] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:00.645154 5495 ts_tablet_manager.cc:1905] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:00.645440 5494 raft_consensus.cc:2229] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:00.645578 5494 raft_consensus.cc:2258] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:00.646000 5494 ts_tablet_manager.cc:1905] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:00.646842 5495 ts_tablet_manager.cc:1918] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:00.646932 5495 log.cc:1196] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/03620734eadc4e899f40b41d7d6f9fe2
I20260924 17:39:00.647228 5495 ts_tablet_manager.cc:1939] T 03620734eadc4e899f40b41d7d6f9fe2 P 454deb72c23f45b08427c975afe228e8: Deleting consensus metadata
I20260924 17:39:00.647666 4993 catalog_manager.cc:4841] TS 454deb72c23f45b08427c975afe228e8 (127.0.98.2:37551): tablet 03620734eadc4e899f40b41d7d6f9fe2 (table test-table [id=1b20ebf035fe4631964504a7714b9faa]) successfully deleted
I20260924 17:39:00.647907 5494 ts_tablet_manager.cc:1918] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:00.647984 5494 log.cc:1196] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/03620734eadc4e899f40b41d7d6f9fe2
I20260924 17:39:00.648221 5494 ts_tablet_manager.cc:1939] T 03620734eadc4e899f40b41d7d6f9fe2 P 299c54fabdc24948999d1c7eaabf6c8b: Deleting consensus metadata
I20260924 17:39:00.648854 4992 catalog_manager.cc:4841] TS 299c54fabdc24948999d1c7eaabf6c8b (127.0.98.3:42293): tablet 03620734eadc4e899f40b41d7d6f9fe2 (table test-table [id=1b20ebf035fe4631964504a7714b9faa]) successfully deleted
W20260924 17:39:00.650619 4990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38461 (error 108)
I20260924 17:39:00.651130 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.650962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38461 (local address 127.0.0.1:52766)
0924 17:39:00.651022 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.651088 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:39:00.651239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5197
W20260924 17:39:00.651244 4990 catalog_manager.cc:4584] TS 32aba1d919384e92b2911f228fc14a17 (127.0.98.1:38461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03620734eadc4e899f40b41d7d6f9fe2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38461: connect: Connection refused (error 111)
I20260924 17:39:00.657306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5328
I20260924 17:39:00.661867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 4975
2026-09-24T17:39:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (724 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 17:39:00.681219 392 mini_oidc.cc:152] Starting JWKS server
I20260924 17:39:00.682289 392 webserver.cc:460] Webserver started at https://127.0.0.1:37769/ using document root <none> and password file <none>
I20260924 17:39:00.682420 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:39:00.682608 392 webserver.cc:460] Webserver started at http://127.0.0.1:44641/ using document root <none> and password file <none>
2026-09-24T17:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:00Z Disabled control of system clock
I20260924 17:39:00.690068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/testchainca.cert with env {}
W20260924 17:39:00.773926 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.774143 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.774166 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.775431 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:00.775478 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.775491 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.775504 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:00.775563 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:39:00.775581 5502 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:39:00.775596 5502 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37769/jwks.json
I20260924 17:39:00.777209 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.777477 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.777721 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:00.782917 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.783053 5502 server_base.cc:1034] running on GCE node
W20260924 17:39:00.782946 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:00.782939 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:00.783382 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:00.783677 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:00.784837 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790271540784820 us; error 28 us; skew 500 ppm
I20260924 17:39:00.786141 5502 webserver.cc:460] Webserver started at http://127.0.98.62:36471/ using document root <none> and password file <none>
I20260924 17:39:00.786381 5502 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:00.786458 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:00.786603 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:00.787750 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "070ff8accae143f8b060e5fe5d1221fe"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "5b9c3e785671f871fa106e70a2cac698"
server_key_iv: "fb4d65e1b9930838bfec516fc21de07f"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.788130 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "070ff8accae143f8b060e5fe5d1221fe"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "5b9c3e785671f871fa106e70a2cac698"
server_key_iv: "fb4d65e1b9930838bfec516fc21de07f"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.789463 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:39:00.790315 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:00.790509 5502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:00.790613 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "070ff8accae143f8b060e5fe5d1221fe"
format_stamp: "Formatted at 2026-09-24 17:39:00 on dist-test-slave-d0pf"
server_key: "5b9c3e785671f871fa106e70a2cac698"
server_key_iv: "fb4d65e1b9930838bfec516fc21de07f"
server_key_version: "encryptionkey@0"
I20260924 17:39:00.790690 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:00.806828 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:00.807240 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:00.807402 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:00.811381 5502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43785
I20260924 17:39:00.811425 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43785 every 8 connection(s)
I20260924 17:39:00.811798 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:00.812824 5569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:00.814687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5502
I20260924 17:39:00.814802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:00.814986 392 external_mini_cluster.cc:1428] Setting key 71b614527c5bd25bd03a445a88e0ecb2
I20260924 17:39:00.815557 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe: Bootstrap starting.
I20260924 17:39:00.816169 5569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:00.816480 5569 log.cc:824] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:00.817363 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe: No bootstrap required, opened a new log
I20260924 17:39:00.819682 5569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260924 17:39:00.819871 5569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:00.819928 5569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070ff8accae143f8b060e5fe5d1221fe, State: Initialized, Role: FOLLOWER
I20260924 17:39:00.820081 5569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260924 17:39:00.820158 5569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:00.820181 5569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:00.820261 5569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:00.820943 5569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260924 17:39:00.821053 5569 leader_election.cc:302] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070ff8accae143f8b060e5fe5d1221fe; no voters:
I20260924 17:39:00.821233 5569 leader_election.cc:288] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:00.821398 5574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:00.821554 5574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [term 1 LEADER]: Becoming Leader. State: Replica: 070ff8accae143f8b060e5fe5d1221fe, State: Running, Role: LEADER
I20260924 17:39:00.821672 5574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260924 17:39:00.821522 5569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:00.821704 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.815600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.0.98.62:43785)
0924 17:39:00.815811 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.815820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.815846 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:39:00.816322 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.816333 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.816413 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.816569 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:00.816800 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.818252 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.819087 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.819786 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.819923 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:00.820739 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:00.820743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:00.820744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:00.820955 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:00.821000 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:00.821016 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:00.821084 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:00.821200 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 17:39:00.821355 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 17:39:00.822075 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070ff8accae143f8b060e5fe5d1221fe. Latest consensus state: current_term: 1 leader_uuid: "070ff8accae143f8b060e5fe5d1221fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260924 17:39:00.822177 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:00.822326 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070ff8accae143f8b060e5fe5d1221fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ff8accae143f8b060e5fe5d1221fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260924 17:39:00.822408 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:00.822606 5581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:00.822711 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:39:00.823135 5581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:00.824651 5581 catalog_manager.cc:1347] Generated new cluster ID: d63c0c7936604824b597640fcf6cc8d6
I20260924 17:39:00.824707 5581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:00.830127 5581 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:00.830862 5581 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:00.835625 5581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070ff8accae143f8b060e5fe5d1221fe: Generated new TSK 0
I20260924 17:39:00.835836 5581 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 17:39:00.862694 5599 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 17:39:00.862818 5599 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 17:39:00.863477 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:00.846872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.0.1:34190)
0924 17:39:00.847141 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:00.847151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:00.847242 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:00.847471 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:00.847472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:00.847474 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:39:00.847537 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:00.847539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.848522 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.848530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:00.849141 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:00.849143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.849637 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.849646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:00.849702 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:00.849726 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:39:00.862985 (+ 13259us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 17:39:00.863122 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 17:39:00.863795 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:00.846948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.62:43785)
0924 17:39:00.847046 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:00.847049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:00.847244 (+ 195us) server_negotiation.cc:407] Connection header received
0924 17:39:00.847325 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:00.847326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:00.847339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:00.847434 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:39:00.847617 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.848409 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.849230 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:00.849550 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:00.849622 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:00.849855 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:39:00.862820 (+ 12965us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 17:39:00.863718 (+ 898us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 17:39:00.863835 5572 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260924 17:39:00.863834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5502
2026-09-24T17:39:00Z chronyd exiting
I20260924 17:39:00.899461 392 test_util.cc:175] -----------------------------------------------
I20260924 17:39:00.899546 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271516406493-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 17:39:00.900291 392 mini_oidc.cc:152] Starting JWKS server
I20260924 17:39:00.900944 392 webserver.cc:460] Webserver started at https://127.0.0.1:42295/ using document root <none> and password file <none>
I20260924 17:39:00.900985 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:39:00.901127 392 webserver.cc:460] Webserver started at http://127.0.0.1:37185/ using document root <none> and password file <none>
2026-09-24T17:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:00Z Disabled control of system clock
I20260924 17:39:00.909400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42383
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/testchainca.cert with env {}
W20260924 17:39:00.992935 5605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:00.993162 5605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:00.993186 5605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:00.994418 5605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:00.994462 5605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:00.994475 5605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:00.994488 5605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:00.994500 5605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:39:00.994540 5605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:39:00.994568 5605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42295/jwks.json
I20260924 17:39:00.996127 5605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:00.996404 5605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:00.996618 5605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:01.002077 5613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:01.002063 5611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:01.002173 5605 server_base.cc:1034] running on GCE node
W20260924 17:39:01.002068 5610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:01.002527 5605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:01.002851 5605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:01.004037 5605 hybrid_clock.cc:648] HybridClock initialized: now 1790271541004006 us; error 58 us; skew 500 ppm
I20260924 17:39:01.005317 5605 webserver.cc:460] Webserver started at http://127.0.98.62:42119/ using document root <none> and password file <none>
I20260924 17:39:01.005565 5605 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:01.005635 5605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:01.005785 5605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:01.006884 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "7e03279f247c4f62b58ab20be26b4c86"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "48f49fc22713a0f04004d0980d215d33"
server_key_iv: "0cd980452a43924b3da80255eeef1581"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.007284 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e03279f247c4f62b58ab20be26b4c86"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "48f49fc22713a0f04004d0980d215d33"
server_key_iv: "0cd980452a43924b3da80255eeef1581"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.008649 5605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:01.009392 5619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:01.009559 5605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:39:01.009663 5605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "7e03279f247c4f62b58ab20be26b4c86"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "48f49fc22713a0f04004d0980d215d33"
server_key_iv: "0cd980452a43924b3da80255eeef1581"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.009739 5605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:01.041088 5605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:01.041476 5605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:01.041646 5605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:01.045686 5605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42383
I20260924 17:39:01.045724 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42383 every 8 connection(s)
I20260924 17:39:01.046087 5605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:01.047115 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:01.049396 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86: Bootstrap starting.
I20260924 17:39:01.049949 5672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:01.050182 5672 log.cc:824] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:01.050870 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86: No bootstrap required, opened a new log
I20260924 17:39:01.052532 5672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:39:01.052676 5672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:01.052698 5672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e03279f247c4f62b58ab20be26b4c86, State: Initialized, Role: FOLLOWER
I20260924 17:39:01.052806 5672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:39:01.052857 5672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:01.052878 5672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:01.052906 5672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:01.053414 5672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:39:01.053508 5672 leader_election.cc:302] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e03279f247c4f62b58ab20be26b4c86; no voters:
I20260924 17:39:01.053637 5672 leader_election.cc:288] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:01.053710 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:01.053872 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [term 1 LEADER]: Becoming Leader. State: Replica: 7e03279f247c4f62b58ab20be26b4c86, State: Running, Role: LEADER
I20260924 17:39:01.053905 5672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:01.053972 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:39:01.054342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5605
I20260924 17:39:01.054458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:01.054692 392 external_mini_cluster.cc:1428] Setting key 62deb5e80d398ada6a2efab2270b7719
I20260924 17:39:01.055500 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e03279f247c4f62b58ab20be26b4c86. Latest consensus state: current_term: 1 leader_uuid: "7e03279f247c4f62b58ab20be26b4c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:39:01.055572 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.055764 5690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:01.055914 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e03279f247c4f62b58ab20be26b4c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e03279f247c4f62b58ab20be26b4c86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:39:01.055990 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.056314 5690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:01.057806 5690 catalog_manager.cc:1347] Generated new cluster ID: f2ee1a25197b4b3ab844549b9aeee83f
I20260924 17:39:01.057857 5690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:01.060985 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.055190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.0.98.62:42383)
0924 17:39:01.055429 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.055438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.055475 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:39:01.055589 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.055597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.055672 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.055820 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:01.056236 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.057609 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.058847 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.059476 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.059592 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.060290 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:01.060293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:01.060294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:01.060493 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:01.060532 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:01.060546 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:01.060605 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:01.060702 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 17:39:01.060777 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260924 17:39:01.061887 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:39:01.066284 5690 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:01.066778 5690 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:01.077885 5690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e03279f247c4f62b58ab20be26b4c86: Generated new TSK 0
I20260924 17:39:01.078123 5690 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 17:39:01.099303 5702 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 17:39:01.099406 5702 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 17:39:01.099736 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.085132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.0.98.62:42383)
0924 17:39:01.085184 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.085186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.085293 (+ 107us) server_negotiation.cc:407] Connection header received
0924 17:39:01.085378 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.085379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.085390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.085456 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:39:01.085602 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.086107 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.087269 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.087501 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.087567 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.087774 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:39:01.099549 (+ 11775us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 17:39:01.099682 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 17:39:01.099822 5689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 17:39:01.099928 5701 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:01.085024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.0.1:40132)
0924 17:39:01.085211 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:01.085223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:01.085295 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:01.085473 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:01.085474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:01.085477 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:39:01.085534 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.085537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.086185 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.086186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.087167 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.087171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.087586 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.087595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.087667 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.087672 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:39:01.099674 (+ 12002us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 17:39:01.099814 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260924 17:39:01.100265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5605
2026-09-24T17:39:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 17:39:01.138090 392 mini_oidc.cc:152] Starting JWKS server
I20260924 17:39:01.138808 392 webserver.cc:460] Webserver started at https://127.0.0.1:40667/ using document root <none> and password file <none>
I20260924 17:39:01.138882 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:39:01.139076 392 webserver.cc:460] Webserver started at http://127.0.0.1:44019/ using document root <none> and password file <none>
2026-09-24T17:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:01Z Disabled control of system clock
I20260924 17:39:01.146741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35177
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35177
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40667/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 17:39:01.226600 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:01.226811 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:01.226832 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:01.228052 5708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:01.228094 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:01.228107 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:01.228120 5708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:01.228132 5708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:39:01.228145 5708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:39:01.228200 5708 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40667/jwks.json
I20260924 17:39:01.229560 5708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40667/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:01.229815 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:01.230103 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:01.234916 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:01.234997 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:01.235004 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:01.235067 5708 server_base.cc:1034] running on GCE node
I20260924 17:39:01.235394 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:01.235694 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:01.236861 5708 hybrid_clock.cc:648] HybridClock initialized: now 1790271541236846 us; error 45 us; skew 500 ppm
I20260924 17:39:01.238082 5708 webserver.cc:460] Webserver started at http://127.0.98.62:44993/ using document root <none> and password file <none>
I20260924 17:39:01.238317 5708 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:01.238384 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:01.238548 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:01.239679 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "8c328f5b0e4a4fee8f6191cceacd93e1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "9e8521b7719d92dee4fb1ebd3fe3b81c"
server_key_iv: "4fb0ef32387438930e72dd2ffd8155e3"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.240052 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c328f5b0e4a4fee8f6191cceacd93e1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "9e8521b7719d92dee4fb1ebd3fe3b81c"
server_key_iv: "4fb0ef32387438930e72dd2ffd8155e3"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.241420 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:39:01.242146 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:01.242352 5708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:01.242447 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "8c328f5b0e4a4fee8f6191cceacd93e1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "9e8521b7719d92dee4fb1ebd3fe3b81c"
server_key_iv: "4fb0ef32387438930e72dd2ffd8155e3"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.242529 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:01.256882 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:01.257254 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:01.257409 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:01.261225 5708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35177
I20260924 17:39:01.261299 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35177 every 8 connection(s)
I20260924 17:39:01.261644 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:01.262333 5775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:01.264959 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1: Bootstrap starting.
I20260924 17:39:01.265558 5775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:01.265856 5775 log.cc:824] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:01.266535 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1: No bootstrap required, opened a new log
I20260924 17:39:01.268115 5775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 17:39:01.268268 5775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:01.268293 5775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c328f5b0e4a4fee8f6191cceacd93e1, State: Initialized, Role: FOLLOWER
I20260924 17:39:01.268493 5775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 17:39:01.268585 5775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:01.268638 5775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:01.268690 5775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:01.269232 5775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 17:39:01.269356 5775 leader_election.cc:302] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c328f5b0e4a4fee8f6191cceacd93e1; no voters:
I20260924 17:39:01.269556 5775 leader_election.cc:288] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:01.269609 5778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:01.269807 5778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c328f5b0e4a4fee8f6191cceacd93e1, State: Running, Role: LEADER
I20260924 17:39:01.269842 5775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:01.269922 5778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 17:39:01.270290 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c328f5b0e4a4fee8f6191cceacd93e1. Latest consensus state: current_term: 1 leader_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } } }
I20260924 17:39:01.270411 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.270506 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c328f5b0e4a4fee8f6191cceacd93e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } } }
I20260924 17:39:01.270599 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.270633 5787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:01.271231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5708
I20260924 17:39:01.271341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:01.271539 392 external_mini_cluster.cc:1428] Setting key b4af0b9d5bb7b8f4ced1349715c99236
I20260924 17:39:01.271847 5787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:01.274701 5787 catalog_manager.cc:1347] Generated new cluster ID: 6c674ed402b24e199fb345a2f77e4cd2
I20260924 17:39:01.274772 5787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:01.277633 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.272010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.98.62:35177)
0924 17:39:01.272425 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.272435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.272467 (+ 32us) server_negotiation.cc:407] Connection header received
0924 17:39:01.272606 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.272614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.272686 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.272831 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:01.273121 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.274533 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.275558 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.276018 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.276159 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.276912 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:01.276916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:01.276916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:01.277105 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:01.277140 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:01.277152 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:01.277249 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:01.277338 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:39:01.277430 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":76,"threads_started":1}
I20260924 17:39:01.278640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:39:01.291754 5787 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:01.292497 5787 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:01.300778 5787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c328f5b0e4a4fee8f6191cceacd93e1: Generated new TSK 0
I20260924 17:39:01.301074 5787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:01.311546 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.307601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.62:35177)
0924 17:39:01.307673 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.307675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.307800 (+ 125us) server_negotiation.cc:407] Connection header received
0924 17:39:01.307889 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.307890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.307902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.307971 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:39:01.308124 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.308648 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.309808 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.310003 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.310070 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.310262 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:39:01.311242 (+ 980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 17:39:01.311458 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 17:39:01.311604 5804 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:01.307535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35177 (local address 127.0.0.1:56878)
0924 17:39:01.307718 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:01.307730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:01.307810 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:01.307989 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:01.307990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:01.307992 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:39:01.308055 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.308057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.308733 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.308734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.309698 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.309701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.310086 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.310094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.310154 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.310160 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:39:01.311359 (+ 1199us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 17:39:01.311487 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35177: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
W20260924 17:39:01.311620 5796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260924 17:39:01.312168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5708
E20260924 17:39:01.324486 5805 webserver.cc:533] Webserver: sslize failed:
E20260924 17:39:01.324581 5805 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T17:39:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T17:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:01Z Disabled control of system clock
I20260924 17:39:01.359386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40965
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40965
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/testchainca.cert with env {}
W20260924 17:39:01.439543 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:01.439749 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:01.439771 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:01.441027 5809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:01.441098 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:01.441113 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:01.441151 5809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:01.441181 5809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:39:01.441201 5809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:39:01.441223 5809 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 17:39:01.443580 5809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:01.443881 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:01.444141 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:01.449047 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:01.449177 5809 server_base.cc:1034] running on GCE node
W20260924 17:39:01.449033 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:01.449100 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:01.449467 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:01.449793 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:01.450963 5809 hybrid_clock.cc:648] HybridClock initialized: now 1790271541450943 us; error 30 us; skew 500 ppm
I20260924 17:39:01.452272 5809 webserver.cc:460] Webserver started at http://127.0.98.62:36825/ using document root <none> and password file <none>
I20260924 17:39:01.452531 5809 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:01.452607 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:01.452754 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:01.453861 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "c54c3ef3c39142d88cf84f67108970f1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "bd028f73ac57f219f92dc9fc4679d1e1"
server_key_iv: "56e43929da73041768b374821d594625"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.454213 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "c54c3ef3c39142d88cf84f67108970f1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "bd028f73ac57f219f92dc9fc4679d1e1"
server_key_iv: "56e43929da73041768b374821d594625"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.455616 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:01.456352 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:01.456526 5809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:01.456631 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "c54c3ef3c39142d88cf84f67108970f1"
format_stamp: "Formatted at 2026-09-24 17:39:01 on dist-test-slave-d0pf"
server_key: "bd028f73ac57f219f92dc9fc4679d1e1"
server_key_iv: "56e43929da73041768b374821d594625"
server_key_version: "encryptionkey@0"
I20260924 17:39:01.456705 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:01.465142 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:01.465437 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:01.465593 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:01.469877 5809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40965
I20260924 17:39:01.469939 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40965 every 8 connection(s)
I20260924 17:39:01.470299 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:01.471303 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:01.473443 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: Bootstrap starting.
I20260924 17:39:01.473455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5809
I20260924 17:39:01.473572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:01.473759 392 external_mini_cluster.cc:1428] Setting key 9728a559867dd833d307e3d66c53fbcb
I20260924 17:39:01.474094 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:01.474431 5876 log.cc:824] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:01.475193 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: No bootstrap required, opened a new log
I20260924 17:39:01.477191 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } }
I20260924 17:39:01.477378 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:01.477423 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54c3ef3c39142d88cf84f67108970f1, State: Initialized, Role: FOLLOWER
I20260924 17:39:01.477560 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } }
I20260924 17:39:01.477645 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:01.477708 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:01.477757 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:01.478386 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } }
I20260924 17:39:01.478487 5876 leader_election.cc:302] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54c3ef3c39142d88cf84f67108970f1; no voters:
I20260924 17:39:01.478654 5876 leader_election.cc:288] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:01.478770 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:01.478931 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:01.478920 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [term 1 LEADER]: Becoming Leader. State: Replica: c54c3ef3c39142d88cf84f67108970f1, State: Running, Role: LEADER
I20260924 17:39:01.479254 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } }
I20260924 17:39:01.479647 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54c3ef3c39142d88cf84f67108970f1. Latest consensus state: current_term: 1 leader_uuid: "c54c3ef3c39142d88cf84f67108970f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } } }
I20260924 17:39:01.479647 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54c3ef3c39142d88cf84f67108970f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c3ef3c39142d88cf84f67108970f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40965 } } }
I20260924 17:39:01.479671 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.474247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.0.98.62:40965)
0924 17:39:01.474469 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.474477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.474501 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:39:01.474953 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.474960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.475016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.475191 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:01.475410 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.476282 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.477427 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.478014 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.478108 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.478880 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:01.478884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:01.478884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:01.479108 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:01.479154 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:01.479171 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:01.479240 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:01.479326 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 17:39:01.479407 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260924 17:39:01.479812 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.480283 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:01.480754 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 17:39:01.481036 5897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:39:01.481097 5897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:39:01.481190 5890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:01.481328 5890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:01.482698 5890 catalog_manager.cc:1347] Generated new cluster ID: 5bca7691b1d54394b470a6106930f521
I20260924 17:39:01.482746 5890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:01.498903 5890 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:01.499496 5890 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:01.506472 5890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54c3ef3c39142d88cf84f67108970f1: Generated new TSK 0
I20260924 17:39:01.506675 5890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:01.615906 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:01.517971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.0.98.62:40965)
0924 17:39:01.518012 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:01.518014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:01.518129 (+ 115us) server_negotiation.cc:407] Connection header received
0924 17:39:01.518206 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:01.518207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:01.518218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:01.518289 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:39:01.518422 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.518915 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.519715 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.519921 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.520008 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.520190 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:39:01.615611 (+ 95421us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 17:39:01.615810 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42778: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 17:39:01.615998 5905 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:01.517881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40965 (local address 127.0.0.1:42778)
0924 17:39:01.518056 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:01.518068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:01.518147 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:01.518290 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:01.518291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:01.518292 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:39:01.518351 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.518354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.519025 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.519026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.519634 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:01.519636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:01.520017 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:01.520023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:01.520077 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:01.520081 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:39:01.615763 (+ 95682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 17:39:01.615890 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
W20260924 17:39:01.616014 5879 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42778: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 17:39:01.616287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5809
2026-09-24T17:39:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (281 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:01 dist-test-slave-d0pf krb5kdc[5917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:01 dist-test-slave-d0pf krb5kdc[5917](info): set up 2 sockets
Sep 24 17:39:01 dist-test-slave-d0pf krb5kdc[5917](info): commencing operation
krb5kdc: starting...
W20260924 17:39:02.839277 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:02 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:39:02.958853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:03.042620 5930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:03.042863 5930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:03.042889 5930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:03.044240 5930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:03.044309 5930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:03.044324 5930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:03.044337 5930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:03.044349 5930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:03.045962 5930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:03.046183 5930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:03.046375 5930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:03.051542 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.051570 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.051542 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.051923 5930 server_base.cc:1034] running on GCE node
I20260924 17:39:03.052217 5930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:03.052587 5930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:03.053778 5930 hybrid_clock.cc:648] HybridClock initialized: now 1790271543053744 us; error 50 us; skew 500 ppm
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.056782 5930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:39:03.057349 5930 webserver.cc:460] Webserver started at http://127.0.98.62:37387/ using document root <none> and password file <none>
I20260924 17:39:03.057631 5930 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:03.057713 5930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:03.057865 5930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:03.059012 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "c28510ebbcc646acacc676c72f81dba0"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "7a78f2e9d48ac72dbbac957e35f44fca"
server_key_iv: "3939a3b9a634485c46db4dae065b444b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.059471 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "c28510ebbcc646acacc676c72f81dba0"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "7a78f2e9d48ac72dbbac957e35f44fca"
server_key_iv: "3939a3b9a634485c46db4dae065b444b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.060988 5930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:03.061873 5945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.062137 5930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:03.062258 5930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "c28510ebbcc646acacc676c72f81dba0"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "7a78f2e9d48ac72dbbac957e35f44fca"
server_key_iv: "3939a3b9a634485c46db4dae065b444b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.062342 5930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:03.074174 5930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:03.076815 5930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:03.077014 5930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:03.081295 5930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40011
I20260924 17:39:03.081332 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40011 every 8 connection(s)
I20260924 17:39:03.081873 5930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:03.082613 5998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:03.083932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 5930
I20260924 17:39:03.084062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:03.084256 392 external_mini_cluster.cc:1428] Setting key 5052d8c3fea0ed079186bf541fde65e0
I20260924 17:39:03.085415 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0: Bootstrap starting.
I20260924 17:39:03.086215 5998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:03.086578 5998 log.cc:824] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:03.087363 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0: No bootstrap required, opened a new log
I20260924 17:39:03.090014 5998 raft_consensus.cc:348] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } }
I20260924 17:39:03.090230 5998 raft_consensus.cc:374] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:03.090273 5998 raft_consensus.cc:729] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28510ebbcc646acacc676c72f81dba0, State: Initialized, Role: FOLLOWER
I20260924 17:39:03.090413 5998 consensus_queue.cc:260] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } }
I20260924 17:39:03.090505 5998 raft_consensus.cc:388] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:03.090544 5998 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:03.090601 5998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:03.091456 5998 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } }
I20260924 17:39:03.091563 5998 leader_election.cc:302] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c28510ebbcc646acacc676c72f81dba0; no voters:
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:03.091806 5998 leader_election.cc:288] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:03.091833 6003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:03.091950 6003 raft_consensus.cc:686] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [term 1 LEADER]: Becoming Leader. State: Replica: c28510ebbcc646acacc676c72f81dba0, State: Running, Role: LEADER
I20260924 17:39:03.092036 6003 consensus_queue.cc:237] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } }
I20260924 17:39:03.092087 5998 sys_catalog.cc:564] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:03.092413 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c28510ebbcc646acacc676c72f81dba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } } }
I20260924 17:39:03.092514 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:03.092722 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28510ebbcc646acacc676c72f81dba0. Latest consensus state: current_term: 1 leader_uuid: "c28510ebbcc646acacc676c72f81dba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28510ebbcc646acacc676c72f81dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40011 } } }
I20260924 17:39:03.092800 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:03.092765 6009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:03.093438 6009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:03.095814 6009 catalog_manager.cc:1347] Generated new cluster ID: 72b9d55e153446f4b64e6abfa0a05bb3
I20260924 17:39:03.095880 6009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:03.097075 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.084835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.0.98.62:40011)
0924 17:39:03.085022 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.085029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.085056 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:39:03.085457 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.085463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.085510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.085681 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.086526 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.087748 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.088850 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.089280 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.089372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.092441 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.092446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.092446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.092461 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.095276 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.095743 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.095746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.095747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.095854 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.096096 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.096097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.096097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.096412 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.096575 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.096738 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.096878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260924 17:39:03.103680 6009 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:03.104472 6009 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:03.109781 6009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c28510ebbcc646acacc676c72f81dba0: Generated new TSK 0
I20260924 17:39:03.110013 6009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:39:03.147668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:03.235698 6024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:03.235922 6024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:03.235948 6024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:03.237180 6024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:03.237228 6024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:03.237257 6024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:03.238832 6024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:03.239125 6024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:03.239392 6024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:03.244746 6030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.244784 6029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.244764 6032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.244746 6024 server_base.cc:1034] running on GCE node
I20260924 17:39:03.245194 6024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:03.245468 6024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:03.246680 6024 hybrid_clock.cc:648] HybridClock initialized: now 1790271543246669 us; error 40 us; skew 500 ppm
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.249562 6024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:39:03.250079 6024 webserver.cc:460] Webserver started at http://127.0.98.1:44601/ using document root <none> and password file <none>
I20260924 17:39:03.250347 6024 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:03.250425 6024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:03.250579 6024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:03.251796 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "f636de5b17434f0381a61ffca34429f3"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "59f6188bf276edb75f2d0efe0c405bc7"
server_key_iv: "bfb20818ba26f08485f9845e2ab553c7"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.252197 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f636de5b17434f0381a61ffca34429f3"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "59f6188bf276edb75f2d0efe0c405bc7"
server_key_iv: "bfb20818ba26f08485f9845e2ab553c7"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.253790 6024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:03.254709 6039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.254902 6024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:03.255015 6024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "f636de5b17434f0381a61ffca34429f3"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "59f6188bf276edb75f2d0efe0c405bc7"
server_key_iv: "bfb20818ba26f08485f9845e2ab553c7"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.255163 6024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:03.266000 6024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:03.268430 6024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:03.268618 6024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:03.268872 6024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:03.269204 6024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:03.269263 6024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.269310 6024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:03.269343 6024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.275422 6024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32809
I20260924 17:39:03.275484 6152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32809 every 8 connection(s)
I20260924 17:39:03.275976 6024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:03.285204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6024
I20260924 17:39:03.285324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:03.285553 392 external_mini_cluster.cc:1428] Setting key 73dc32a1d85cc79d750724d4266a71ed
I20260924 17:39:03.288305 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.276847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47283 (local address 127.0.98.62:40011)
0924 17:39:03.276891 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.276894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.277080 (+ 186us) server_negotiation.cc:407] Connection header received
0924 17:39:03.278069 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.278070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.278080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.278149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.278803 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.279443 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.280474 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.280691 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.280751 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.283841 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.283843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.283843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.283853 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.286353 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.286997 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.286998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.286999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.287096 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.287424 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.287425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.287426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.287604 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.287685 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.288099 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.288234 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:03.288530 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.276795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40011 (local address 127.0.98.1:47283)
0924 17:39:03.276961 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:03.277005 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:03.277963 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:03.278208 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:03.278211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:03.278485 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:03.278690 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.278695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.279557 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.279558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.280387 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.280389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.280788 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.280789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.280908 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.281403 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:03.281412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:03.283721 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:03.286471 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.286483 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.286486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.286865 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.287178 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.287179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.287179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.287352 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.287685 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:03.287692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:03.287893 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 17:39:03.288037 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 17:39:03.288243 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260924 17:39:03.289047 6153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40011
I20260924 17:39:03.289151 6153 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:03.289374 6153 heartbeater.cc:511] Master 127.0.98.62:40011 requested a full tablet report, sending...
I20260924 17:39:03.289994 5962 ts_manager.cc:194] Registered new tserver with Master: f636de5b17434f0381a61ffca34429f3 (127.0.98.1:32809)
I20260924 17:39:03.290956 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47283
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:39:03.314646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:03.398727 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:03.398948 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:03.398973 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:03.400564 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:03.400626 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:03.400656 6158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:03.402343 6158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:03.402632 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:03.402875 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:03.408226 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.408238 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.408392 6158 server_base.cc:1034] running on GCE node
W20260924 17:39:03.408458 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.408687 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:03.409013 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:03.410192 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790271543410181 us; error 42 us; skew 500 ppm
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.412983 6158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:39:03.413568 6158 webserver.cc:460] Webserver started at http://127.0.98.2:35243/ using document root <none> and password file <none>
I20260924 17:39:03.413826 6158 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:03.413898 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:03.414052 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:03.415269 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "52f24b33a3f6480c8a984d07b4b2f0f8"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b3895a8c9c42dc20064f3d541d49f7d1"
server_key_iv: "927eeb6eb68384cdbf103a0b27fe5311"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.415664 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52f24b33a3f6480c8a984d07b4b2f0f8"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b3895a8c9c42dc20064f3d541d49f7d1"
server_key_iv: "927eeb6eb68384cdbf103a0b27fe5311"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.417090 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:03.418036 6173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.418255 6158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:03.418378 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "52f24b33a3f6480c8a984d07b4b2f0f8"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b3895a8c9c42dc20064f3d541d49f7d1"
server_key_iv: "927eeb6eb68384cdbf103a0b27fe5311"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.418465 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:03.442931 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:03.445382 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:03.445574 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:03.445832 6158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:03.446159 6158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:03.446234 6158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.446300 6158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:03.446328 6158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.453332 6158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36625
I20260924 17:39:03.453372 6286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36625 every 8 connection(s)
I20260924 17:39:03.453738 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:03.460119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6158
I20260924 17:39:03.460210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:03.460395 392 external_mini_cluster.cc:1428] Setting key 99a370a6b668f60a2c65177e3763ddfb
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:03.464313 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.454520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51071 (local address 127.0.98.62:40011)
0924 17:39:03.454591 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.454593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.454842 (+ 249us) server_negotiation.cc:407] Connection header received
0924 17:39:03.455910 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.455911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.455924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.455976 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.456648 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.457152 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.458308 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.458508 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.458594 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.461100 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.461102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.461102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.461111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.462738 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.463268 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.463270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.463270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.463341 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.463616 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.463617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.463618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.463752 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.463833 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.464127 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.464212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:39:03.464568 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.454551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40011 (local address 127.0.98.2:51071)
0924 17:39:03.454716 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:03.454753 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:03.455784 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:03.456053 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:03.456057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:03.456335 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:03.456534 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.456538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.457255 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.457257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.458221 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.458223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.458604 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.458605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.458749 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.459377 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:03.459386 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:03.460988 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:03.462849 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.462866 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.462869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.463167 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.463413 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.463414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.463415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.463537 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.463824 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:03.463830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:03.463985 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 17:39:03.464092 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 17:39:03.464282 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260924 17:39:03.465107 6287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40011
I20260924 17:39:03.465222 6287 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:03.465402 6287 heartbeater.cc:511] Master 127.0.98.62:40011 requested a full tablet report, sending...
I20260924 17:39:03.465735 5962 ts_manager.cc:194] Registered new tserver with Master: 52f24b33a3f6480c8a984d07b4b2f0f8 (127.0.98.2:36625)
I20260924 17:39:03.466188 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51071
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:39:03.490314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:03.581951 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:03.582430 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:03.582464 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:03.583988 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:03.584048 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:03.584079 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:03.585829 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:03.586138 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:03.586393 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:03.592049 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:03.592049 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.592288 6292 server_base.cc:1034] running on GCE node
W20260924 17:39:03.592053 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:03.592725 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:03.593044 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:03.594247 6292 hybrid_clock.cc:648] HybridClock initialized: now 1790271543594224 us; error 45 us; skew 500 ppm
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.597635 6292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:39:03.598313 6292 webserver.cc:460] Webserver started at http://127.0.98.3:46189/ using document root <none> and password file <none>
I20260924 17:39:03.598572 6292 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:03.598623 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:03.598795 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:03.599989 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "21f94c38e9d4428a9bc4dcf95d0c84ae"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b776830849e148e086fd5515c9242ada"
server_key_iv: "2edd12b7759b5d29e03edfc888d68b2b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.600373 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21f94c38e9d4428a9bc4dcf95d0c84ae"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b776830849e148e086fd5515c9242ada"
server_key_iv: "2edd12b7759b5d29e03edfc888d68b2b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.601958 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:03.602970 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.603276 6292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:03.603366 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "21f94c38e9d4428a9bc4dcf95d0c84ae"
format_stamp: "Formatted at 2026-09-24 17:39:03 on dist-test-slave-d0pf"
server_key: "b776830849e148e086fd5515c9242ada"
server_key_iv: "2edd12b7759b5d29e03edfc888d68b2b"
server_key_version: "encryptionkey@0"
I20260924 17:39:03.603473 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:03.613967 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:03.616916 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:03.617084 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:03.617362 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:03.617723 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:03.617764 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.617789 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:03.617801 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:03.624215 6292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35627
I20260924 17:39:03.624346 6420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35627 every 8 connection(s)
I20260924 17:39:03.624660 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:03.626156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6292
I20260924 17:39:03.626272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:03.626452 392 external_mini_cluster.cc:1428] Setting key 9d5ca92263cb62caacd77f3fe30e00f0
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:03.636888 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.625868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51255 (local address 127.0.98.62:40011)
0924 17:39:03.625964 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.625966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.626016 (+ 50us) server_negotiation.cc:407] Connection header received
0924 17:39:03.627233 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.627234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.627247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.627300 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.628124 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.628630 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.629776 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.629985 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.630053 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.633598 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.633600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.633600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.633608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.635002 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.635544 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.635545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.635545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.635606 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.636061 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.636062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.636063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.636203 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.636271 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.636576 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.636814 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 17:39:03.636935 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.625549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40011 (local address 127.0.98.3:51255)
0924 17:39:03.625779 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:03.625865 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:03.627113 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:03.627380 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:03.627385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:03.627739 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:03.627959 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.627964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.628771 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.628772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.629650 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:03.629652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.630094 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.630095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:03.630231 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.631874 (+ 1643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:03.631884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:03.633473 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:03.635191 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.635200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.635203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.635465 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.635807 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:03.635808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:03.635808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:03.635960 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:03.636299 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:03.636306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:03.636450 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 17:39:03.636549 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 17:39:03.636742 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260924 17:39:03.637786 6421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40011
I20260924 17:39:03.637915 6421 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:03.638139 6421 heartbeater.cc:511] Master 127.0.98.62:40011 requested a full tablet report, sending...
I20260924 17:39:03.638541 5962 ts_manager.cc:194] Registered new tserver with Master: 21f94c38e9d4428a9bc4dcf95d0c84ae (127.0.98.3:35627)
I20260924 17:39:03.638948 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51255
I20260924 17:39:03.649420 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/test-admin.keytab.
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.668372 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 17:39:03.670177 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:03.669261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.1:32809)
0924 17:39:03.669471 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.669475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.669488 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:39:03.669516 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.669518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.669533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.669621 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.670077 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41388: BlockingRecv error: recv got EOF from 127.0.0.1:41388 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
W20260924 17:39:03.670269 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.671428 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.672509 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:39:03.681710 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.673100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.98.1:32809)
0924 17:39:03.673170 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.673172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.673207 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:39:03.673268 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.673269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.673278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.673334 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.673748 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.674474 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.675289 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.675671 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.675739 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.678224 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.678243 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.678243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.678251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.680152 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.680632 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.680635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.680635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.680701 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.681035 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.681036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.681036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.681207 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.681445 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.681525 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.681606 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 17:39:03.684489 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:03.683834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.0.98.1:32809)
0924 17:39:03.683914 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.683916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.683925 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:03.684023 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.684024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.684044 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.684109 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.684438 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41404: BlockingRecv error: recv got EOF from 127.0.0.1:41404 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 17:39:03.684852 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.685988 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.687150 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:39:03.696028 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.687918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.98.1:32809)
0924 17:39:03.687990 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.687992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.688003 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:03.688036 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.688037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.688047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.688103 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.688523 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.689052 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.689814 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.690020 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.690086 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.692958 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.692960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.692960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.692969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.694575 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.694969 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.694970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.694971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.695081 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.695350 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.695352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.695352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.695479 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.695603 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.695809 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.695890 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 17:39:03.697768 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:03.697114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.1:32809)
0924 17:39:03.697175 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.697177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.697219 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:39:03.697297 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.697298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.697308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.697379 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.697678 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41424: BlockingRecv error: recv got EOF from 127.0.0.1:41424 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 17:39:03.697957 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271516406493-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.699021 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:03.699994 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 17:39:03.708438 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:03.700531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.0.98.1:32809)
0924 17:39:03.700625 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:03.700627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:03.700637 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:03.700700 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:03.700701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:03.700710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:03.700784 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:03.701176 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.701682 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.702410 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:03.702617 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:03.702696 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:03.705455 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:03.705457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:03.705457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:03.705465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:03.706939 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.707410 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.707412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.707412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.707507 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:03.707860 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:03.707861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:03.707862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:03.707976 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:03.708069 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:03.708295 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 17:39:03.708371 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 17:39:03.709362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6024
I20260924 17:39:03.714433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6158
I20260924 17:39:03.718812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6292
I20260924 17:39:03.723448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 5930
2026-09-24T17:39:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1936 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[6441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[6441](info): set up 2 sockets
Sep 24 17:39:03 dist-test-slave-d0pf krb5kdc[6441](info): commencing operation
krb5kdc: starting...
W20260924 17:39:04.783363 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:04 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 17:39:04.902740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:04.984992 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:04.985273 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:04.985313 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:04.986570 6454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:04.986652 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:04.986677 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:04.986701 6454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:04.986727 6454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:04.988359 6454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:04.988679 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:04.989048 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:04.994325 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:04.994325 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:04.994464 6454 server_base.cc:1034] running on GCE node
W20260924 17:39:04.994380 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:04.994854 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:04.995220 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:04.996411 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790271544996396 us; error 45 us; skew 500 ppm
Sep 24 17:39:04 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.003466 6454 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 17:39:05.003988 6454 webserver.cc:460] Webserver started at http://127.0.98.62:40005/ using document root <none> and password file <none>
I20260924 17:39:05.004258 6454 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.004333 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.004490 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:05.005636 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "a57e962e8d03421597cf7b8e34bf4715"
format_stamp: "Formatted at 2026-09-24 17:39:04 on dist-test-slave-d0pf"
server_key: "afad7c0209ba9d1edc6000d64c2033ec"
server_key_iv: "5dfe4f6d51cc192da350ce305df95a9c"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.006021 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "a57e962e8d03421597cf7b8e34bf4715"
format_stamp: "Formatted at 2026-09-24 17:39:04 on dist-test-slave-d0pf"
server_key: "afad7c0209ba9d1edc6000d64c2033ec"
server_key_iv: "5dfe4f6d51cc192da350ce305df95a9c"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.007541 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:39:05.008427 6469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.008668 6454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:05.008791 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "a57e962e8d03421597cf7b8e34bf4715"
format_stamp: "Formatted at 2026-09-24 17:39:04 on dist-test-slave-d0pf"
server_key: "afad7c0209ba9d1edc6000d64c2033ec"
server_key_iv: "5dfe4f6d51cc192da350ce305df95a9c"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.008888 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.021520 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.023877 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:05.024055 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.028294 6454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39945
I20260924 17:39:05.028342 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39945 every 8 connection(s)
I20260924 17:39:05.028697 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:05.029788 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:05.032248 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Bootstrap starting.
I20260924 17:39:05.032868 6522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:05.033172 6522 log.cc:824] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:05.033944 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: No bootstrap required, opened a new log
I20260924 17:39:05.035679 6522 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260924 17:39:05.035833 6522 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:05.035857 6522 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57e962e8d03421597cf7b8e34bf4715, State: Initialized, Role: FOLLOWER
I20260924 17:39:05.036034 6522 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260924 17:39:05.036119 6522 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:05.036164 6522 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:05.036216 6522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:05.037058 6522 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260924 17:39:05.037173 6522 leader_election.cc:302] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57e962e8d03421597cf7b8e34bf4715; no voters:
I20260924 17:39:05.037343 6522 leader_election.cc:288] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:05.037429 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:05.037599 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 LEADER]: Becoming Leader. State: Replica: a57e962e8d03421597cf7b8e34bf4715, State: Running, Role: LEADER
I20260924 17:39:05.037650 6522 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:05.037729 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260924 17:39:05.037845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6454
I20260924 17:39:05.037966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:05.038069 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57e962e8d03421597cf7b8e34bf4715. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } }
I20260924 17:39:05.038173 392 external_mini_cluster.cc:1428] Setting key 858756282390b734f64a2afc660a19c6
I20260924 17:39:05.038172 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:05.038358 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } }
I20260924 17:39:05.038429 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:05.038388 6532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:05.039381 6532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:05.042299 6532 catalog_manager.cc:1347] Generated new cluster ID: 1327fdf0142145b3988bf9b3650c3d55
I20260924 17:39:05.042371 6532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:05.051280 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.038853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.0.98.62:39945)
0924 17:39:05.039083 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.039095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.039136 (+ 41us) server_negotiation.cc:407] Connection header received
0924 17:39:05.039286 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.039296 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.039361 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.040293 (+ 932us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.040391 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.041769 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.042897 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.043718 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.043882 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.047198 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.047204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.047205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.047221 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.049433 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.049866 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.049869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.049870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.050021 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.050228 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.050229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.050229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.050594 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.050756 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.050865 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.051018 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260924 17:39:05.052379 6532 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:05.052881 6532 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:05.061209 6532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Generated new TSK 0
I20260924 17:39:05.061446 6532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 17:39:05.097972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:05.186414 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:05.186654 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:05.186697 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:05.187925 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:05.187995 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:05.188043 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:05.189535 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:05.189805 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:05.190039 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:05.195040 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.195022 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.195026 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:05.195252 6548 server_base.cc:1034] running on GCE node
I20260924 17:39:05.195459 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:05.195756 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:05.196945 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790271545196935 us; error 37 us; skew 500 ppm
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.199530 6548 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 17:39:05.200004 6548 webserver.cc:460] Webserver started at http://127.0.98.1:34591/ using document root <none> and password file <none>
I20260924 17:39:05.200259 6548 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.200331 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.200489 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:05.201577 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ecb344160a643a39efa254fc861e47c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "22480c1e7777799540c58fa310ede455"
server_key_iv: "c1e5bf21347ef2ae0a46f4eb87c4cdb8"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.202116 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ecb344160a643a39efa254fc861e47c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "22480c1e7777799540c58fa310ede455"
server_key_iv: "c1e5bf21347ef2ae0a46f4eb87c4cdb8"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.203740 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:05.204680 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.204996 6548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:05.205106 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "7ecb344160a643a39efa254fc861e47c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "22480c1e7777799540c58fa310ede455"
server_key_iv: "c1e5bf21347ef2ae0a46f4eb87c4cdb8"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.205184 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.225402 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.227413 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:05.227586 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.227845 6548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:05.228165 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:05.228219 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.228260 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:05.228308 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.233978 6548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42723
I20260924 17:39:05.234086 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42723 every 8 connection(s)
I20260924 17:39:05.234393 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:05.243909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6548
I20260924 17:39:05.244014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:05.244226 392 external_mini_cluster.cc:1428] Setting key 086226345d5d53bf6aefa5893ac7ce7f
I20260924 17:39:05.244225 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.235139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51707 (local address 127.0.98.62:39945)
0924 17:39:05.235187 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.235189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.235406 (+ 217us) server_negotiation.cc:407] Connection header received
0924 17:39:05.235941 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.235942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.235960 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.236018 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.237000 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.237518 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.238672 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.238962 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.239068 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.241313 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.241315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.241315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.241323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.242634 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.243075 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.243076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.243077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.243140 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.243396 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.243397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.243397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.243530 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.243608 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.244014 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.244123 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:39:05.244449 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.235135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:51707)
0924 17:39:05.235282 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.235318 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.235831 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.236096 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.236100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.236682 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.236876 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.236882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.237659 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.237660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.238588 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.238591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.239073 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.239074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.239210 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.239825 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.239834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.241211 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.242732 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.242739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.242742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.242979 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.243215 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.243216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.243216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.243329 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.243598 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.243604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.243796 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.243937 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.244132 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260924 17:39:05.244971 6677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:05.245085 6677 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:05.245283 6677 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:05.245877 6485 ts_manager.cc:194] Registered new tserver with Master: 7ecb344160a643a39efa254fc861e47c (127.0.98.1:42723)
I20260924 17:39:05.246874 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51707
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 17:39:05.271942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:05.354553 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:05.354810 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:05.354851 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:05.356089 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:05.356169 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:05.356215 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:05.357775 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:05.358062 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:05.358299 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:05.363684 6682 server_base.cc:1034] running on GCE node
W20260924 17:39:05.363715 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.363653 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.363653 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:05.364199 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:05.364485 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:05.365651 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790271545365639 us; error 31 us; skew 500 ppm
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.368199 6682 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 17:39:05.368687 6682 webserver.cc:460] Webserver started at http://127.0.98.2:39583/ using document root <none> and password file <none>
I20260924 17:39:05.368938 6682 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.369007 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.369153 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:05.370237 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "46f41884be83c3c8929bbe3021cba9e8"
server_key_iv: "78a17a9a05f7c73f8c4c36f9072c6770"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.370597 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "46f41884be83c3c8929bbe3021cba9e8"
server_key_iv: "78a17a9a05f7c73f8c4c36f9072c6770"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.372077 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:39:05.372828 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.373016 6682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.373126 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "46f41884be83c3c8929bbe3021cba9e8"
server_key_iv: "78a17a9a05f7c73f8c4c36f9072c6770"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.373202 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.389953 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.392406 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:05.392580 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.392829 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:05.393152 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:05.393210 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.393282 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:05.393317 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.399276 6682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36183
I20260924 17:39:05.399317 6810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36183 every 8 connection(s)
I20260924 17:39:05.399643 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:05.407096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6682
I20260924 17:39:05.407214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:05.407454 392 external_mini_cluster.cc:1428] Setting key 6cde32ae94a9e9e2b8b1941a0be183c2
I20260924 17:39:05.409647 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.400360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56533 (local address 127.0.98.62:39945)
0924 17:39:05.400425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.400427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.400634 (+ 207us) server_negotiation.cc:407] Connection header received
0924 17:39:05.401148 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.401149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.401158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.401218 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.402491 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.402980 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.404034 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.404217 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.404271 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.406510 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.406512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.406512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.406519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.408070 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.408602 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.408603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.408603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.408673 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.408938 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.408939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.408940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.409079 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.409162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.409469 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.409556 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:05.409960 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.400339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:56533)
0924 17:39:05.400502 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.400557 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.401064 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.401287 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.401291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.402190 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.402389 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.402395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.403124 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.403125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.403956 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.403958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.404291 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.404291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.404393 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.404996 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.405005 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.406420 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.408138 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.408145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.408147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.408506 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.408744 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.408745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.408745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.408866 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.409166 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.409172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.409318 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.409439 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.409606 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260924 17:39:05.410490 6811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:05.410602 6811 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:05.410811 6811 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:05.411185 6485 ts_manager.cc:194] Registered new tserver with Master: 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183)
I20260924 17:39:05.411618 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56533
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 17:39:05.433769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:05.515926 6816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:05.516156 6816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:05.516181 6816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:05.517386 6816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:05.517431 6816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:05.517460 6816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:05.518937 6816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:05.519230 6816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:05.519484 6816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:05.524494 6824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.524507 6821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.524505 6822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:05.524708 6816 server_base.cc:1034] running on GCE node
I20260924 17:39:05.524957 6816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:05.525216 6816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:05.526366 6816 hybrid_clock.cc:648] HybridClock initialized: now 1790271545526353 us; error 29 us; skew 500 ppm
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.528987 6816 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 17:39:05.529428 6816 webserver.cc:460] Webserver started at http://127.0.98.3:36639/ using document root <none> and password file <none>
I20260924 17:39:05.529651 6816 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.529718 6816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.529871 6816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:05.531133 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e63bb4cad2449489560ab8e63f4e588"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "1397c291cc2fdf67f2be26a7796d7387"
server_key_iv: "c320fc201700eb764cbd3b81eb09ce04"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.531498 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e63bb4cad2449489560ab8e63f4e588"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "1397c291cc2fdf67f2be26a7796d7387"
server_key_iv: "c320fc201700eb764cbd3b81eb09ce04"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.532853 6816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:05.533614 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.533874 6816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:05.533980 6816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "3e63bb4cad2449489560ab8e63f4e588"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "1397c291cc2fdf67f2be26a7796d7387"
server_key_iv: "c320fc201700eb764cbd3b81eb09ce04"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.534057 6816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.553362 6816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.555838 6816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:05.556008 6816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.556252 6816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:05.556572 6816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:05.556625 6816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.556667 6816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:05.556691 6816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.562928 6816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45857
I20260924 17:39:05.562981 6944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45857 every 8 connection(s)
I20260924 17:39:05.563341 6816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:05.568766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6816
I20260924 17:39:05.568847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:05.569051 392 external_mini_cluster.cc:1428] Setting key 39bde8bbe605f54dd8940c8d534759ad
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:05.572718 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.564051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40795 (local address 127.0.98.62:39945)
0924 17:39:05.564129 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.564131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.564334 (+ 203us) server_negotiation.cc:407] Connection header received
0924 17:39:05.564825 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.564826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.564837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.564892 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.565788 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.566268 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.567270 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.567452 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.567507 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.569943 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.569944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.569945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.569962 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.571445 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.571794 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.571795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.571795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.571867 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.572130 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.572131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.572131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.572237 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.572288 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.572584 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.572649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:39:05.572937 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.564019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:40795)
0924 17:39:05.564187 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.564223 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.564739 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.564944 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.564948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.565514 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.565711 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.565715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.566349 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.566350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.567197 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.567199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.567523 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.567524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.567618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.568096 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.568103 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.569854 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.571512 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.571517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.571519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.571731 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.571955 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.571956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.571956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.572067 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.572314 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.572320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.572457 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.572554 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.572679 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260924 17:39:05.573403 6945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:05.573493 6945 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:05.573634 6945 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:05.573944 6485 ts_manager.cc:194] Registered new tserver with Master: 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:05.574275 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40795
I20260924 17:39:05.580899 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 17:39:05.608201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33309
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39945,127.0.98.61:33309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:05.690433 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:05.690656 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:05.690678 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:05.691797 6950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:05.691833 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:05.691846 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:05.691859 6950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:05.691871 6950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:05.693365 6950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39945,127.0.98.61:33309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
W20260924 17:39:05.693470 6950 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39945,127.0.98.61:33309'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 17:39:05.693580 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:05.693745 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:05.698545 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.698531 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.698534 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:05.698855 6950 server_base.cc:1034] running on GCE node
I20260924 17:39:05.699110 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:05.699411 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:05.700577 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790271545700563 us; error 26 us; skew 500 ppm
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.703181 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 17:39:05.703677 6950 webserver.cc:460] Webserver started at http://127.0.98.61:37285/ using document root <none> and password file <none>
I20260924 17:39:05.703912 6950 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.703980 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.704123 6950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:05.705269 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/instance:
uuid: "be6c68f5902a4725af474b575d30b50c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "df988ac1c75d75ccdee7ab2a4573dcd2"
server_key_iv: "c163762bde9c9d650f1507e43d4d2b2f"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.705646 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal/instance:
uuid: "be6c68f5902a4725af474b575d30b50c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "df988ac1c75d75ccdee7ab2a4573dcd2"
server_key_iv: "c163762bde9c9d650f1507e43d4d2b2f"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.707077 6950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:05.707885 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.708091 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:05.708204 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "be6c68f5902a4725af474b575d30b50c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "df988ac1c75d75ccdee7ab2a4573dcd2"
server_key_iv: "c163762bde9c9d650f1507e43d4d2b2f"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.708272 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.720777 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.722772 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:05.722944 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.727495 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33309
I20260924 17:39:05.727537 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33309 every 8 connection(s)
I20260924 17:39:05.727897 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
I20260924 17:39:05.728871 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:05.730512 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:39:05.733232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 6950
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:05.741519 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.731352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.0.98.62:39945)
0924 17:39:05.731445 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.731448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.731629 (+ 181us) server_negotiation.cc:407] Connection header received
0924 17:39:05.732346 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.732347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.732360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.732417 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.733089 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.733693 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.735261 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.735587 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.735668 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.738386 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.738388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.738389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.738397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.739982 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.740474 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.740475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.740476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.740543 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.740786 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.740787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.740787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.741006 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.741078 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.741348 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.741427 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:39:05.741575 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.731313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42322)
0924 17:39:05.731504 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.731542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.732206 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.732499 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.732503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.732759 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.732966 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.732970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.733798 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.733799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.735183 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.735185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.735676 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.735677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.735786 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.736326 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.736337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.738270 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.740067 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.740072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.740074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.740395 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.740613 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.740614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.740614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.740717 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.741078 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.741083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.741203 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.741290 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.741413 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260924 17:39:05.742245 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:39:05.743547 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.733777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.98.61:33309)
0924 17:39:05.734019 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.734024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.734038 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:39:05.734056 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.734058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.734074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.734139 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.734893 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.735902 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.736921 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.737389 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.737470 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.740002 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.740010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.740010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.740017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.742217 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.742642 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.742645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.742646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.742705 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.742924 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.742925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.742925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.743137 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.743322 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.743366 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.743446 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":159,"threads_started":1}
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:05.749333 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.742568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.0.98.61:33309)
0924 17:39:05.742717 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.742718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.742724 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:39:05.742739 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.742739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.742748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.742804 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.743126 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.743827 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.744536 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.744745 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.744800 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.746471 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.746472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.746472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.746478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.748377 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.748683 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.748684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.748684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.748733 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.748898 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.748899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.748899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.749009 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.749058 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.749182 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.749244 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":105,"threads_started":1}
I20260924 17:39:05.749388 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.742516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56188)
0924 17:39:05.742567 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.742578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.742615 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.742816 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.742817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.742959 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.743002 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.743003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.743904 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.743905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.744471 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.744473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.744797 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.744798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.744843 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.745263 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.745267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.746383 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.748455 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.748456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.748456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.748616 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.748791 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.748792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.748792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.748843 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.749068 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.749068 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.749104 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.749165 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.749289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":51}
I20260924 17:39:05.750702 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Bootstrap starting.
I20260924 17:39:05.751426 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:05.751705 7018 log.cc:824] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:05.752419 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: No bootstrap required, opened a new log
I20260924 17:39:05.753809 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } }
I20260924 17:39:05.753940 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:05.753986 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6c68f5902a4725af474b575d30b50c, State: Initialized, Role: FOLLOWER
I20260924 17:39:05.754125 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } }
I20260924 17:39:05.754431 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:05.754387 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } }
I20260924 17:39:05.754518 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:39:05.756145 7036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:39:05.756217 7036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:39:05.757815 6950 master_runner.cc:428] Detected that this master be6c68f5902a4725af474b575d30b50c is joining an existing cluster
I20260924 17:39:05.757855 6950 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33309
I20260924 17:39:05.766009 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.758882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56196)
0924 17:39:05.759125 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.759135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.759187 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.759338 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.759338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.759535 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.759591 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.759593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.760480 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.760481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.761354 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.761366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.761772 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.761773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.761835 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.762318 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.762323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.762748 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.764722 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.764723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.764724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.765070 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.765282 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.765283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.765283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.765338 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.765579 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.765580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.765617 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.765715 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.765876 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260924 17:39:05.766078 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.759082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.0.98.61:33309)
0924 17:39:05.759116 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.759118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.759239 (+ 121us) server_negotiation.cc:407] Connection header received
0924 17:39:05.759255 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.759256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.759264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.759305 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.759654 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.760398 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.761449 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.761657 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.761757 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.763071 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.763072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.763073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.763079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.764625 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.765162 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.765163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.765164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.765218 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.765409 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.765410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.765410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.765516 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.765588 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.765735 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.765956 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":19}
I20260924 17:39:05.767264 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.758906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.0.98.62:39945)
0924 17:39:05.758954 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.758957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.759109 (+ 152us) server_negotiation.cc:407] Connection header received
0924 17:39:05.759193 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.759193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.759204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.759279 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.759627 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.760398 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.762432 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.762742 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.762820 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.763987 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.763988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.763988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.763995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.765538 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.766063 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.766064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.766064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.766122 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.766767 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.766768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.766768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.766879 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.766937 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.767133 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.767204 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:39:05.767305 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.758854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42332)
0924 17:39:05.759023 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.759054 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.759136 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.759282 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.759282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.759468 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.759539 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.759541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.761476 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.761476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.762330 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.762333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.762828 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.762829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.762894 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.763435 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.763440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.763878 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.765629 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.765631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.765631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.765977 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.766626 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.766627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.766627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.766695 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.766953 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.766954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.767007 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.767104 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.767202 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:05.777258 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.769495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56206)
0924 17:39:05.769643 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.769687 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.769761 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.769899 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.769902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.770248 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.770314 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.770316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.770899 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.770900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.771893 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.771896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.772221 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.772222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.772306 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.772740 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.772745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.774317 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.776001 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.776010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.776012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.776297 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.776516 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.776517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.776518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.776610 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.776937 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.776945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.777003 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.777075 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.777183 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260924 17:39:05.777298 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.769584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.98.61:33309)
0924 17:39:05.769627 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.769629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.769754 (+ 125us) server_negotiation.cc:407] Connection header received
0924 17:39:05.769832 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.769833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.769842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.769885 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.770388 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.770821 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.771972 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.772150 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.772204 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.774411 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.774413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.774413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.774420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.775917 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.776392 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.776392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.776393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.776447 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.776688 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.776689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.776689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.776870 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.776930 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.777122 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.777204 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:39:05.778228 6485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33309: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "be6c68f5902a4725af474b575d30b50c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33309
}
attrs {
promote: true
}
}
dest_uuid: "a57e962e8d03421597cf7b8e34bf4715"
cas_config_opid_index: -1
I20260924 17:39:05.778589 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } }
I20260924 17:39:05.779175 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER be6c68f5902a4725af474b575d30b50c (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:05.779501 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:05.780318 7045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33309
I20260924 17:39:05.780435 6992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 17:39:05.780550 6950 master.cc:537] Master@127.0.98.61:33309 shutting down...
I20260924 17:39:05.780776 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:05.780828 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:05.780933 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
W20260924 17:39:05.781694 6472 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 17:39:05.782507 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 -> Peer be6c68f5902a4725af474b575d30b50c (127.0.98.61:33309): Couldn't send request to peer be6c68f5902a4725af474b575d30b50c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 17:39:05.783303 6950 raft_consensus.cc:2229] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:05.783399 6950 raft_consensus.cc:2258] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:05.783447 6950 tablet_replica.cc:330] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: stopping tablet replica
I20260924 17:39:05.795951 6950 master.cc:552] Master@127.0.98.61:33309 shutdown complete.
I20260924 17:39:05.796026 6950 master_runner.cc:305] Clearing existing system tablet
I20260924 17:39:05.796157 6950 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.796216 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.796841 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:39:05.797408 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.797569 6950 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.797677 6950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:05.797709 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "be6c68f5902a4725af474b575d30b50c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "df988ac1c75d75ccdee7ab2a4573dcd2"
server_key_iv: "c163762bde9c9d650f1507e43d4d2b2f"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.798527 6950 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:05.800420 6950 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 17:39:05.800513 6950 log.cc:1196] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 17:39:05.800717 6950 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Deleting consensus metadata
I20260924 17:39:05.800840 6950 master_runner.cc:315] Copying system tablet from 127.0.98.62:39945
I20260924 17:39:05.801463 6950 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39945
I20260924 17:39:05.808310 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.801805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42334)
0924 17:39:05.801987 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:05.801999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:05.802064 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:05.802221 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:05.802222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:05.802388 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:05.802446 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.802448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.803413 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.803414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.804003 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:05.804006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.804483 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.804483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:05.804533 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.805098 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:05.805103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:05.805426 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:05.807273 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.807274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.807274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.807458 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.807664 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:05.807665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:05.807665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:05.807722 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:05.808048 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:05.808048 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:05.808088 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 17:39:05.808142 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 17:39:05.808227 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260924 17:39:05.808362 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.801916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.98.62:39945)
0924 17:39:05.801992 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.801994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.802053 (+ 59us) server_negotiation.cc:407] Connection header received
0924 17:39:05.802139 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.802140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.802152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.802218 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.802518 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.803320 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.804094 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.804405 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.804487 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.805523 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.805524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.805525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.805532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.807178 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.807530 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.807531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.807532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.807598 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.807843 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.807844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.807844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.807983 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.808054 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.808185 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.808263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:39:05.809060 6506 tablet_copy_service.cc:140] P a57e962e8d03421597cf7b8e34bf4715: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer be6c68f5902a4725af474b575d30b50c ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42334)
I20260924 17:39:05.809154 6506 tablet_copy_service.cc:161] P a57e962e8d03421597cf7b8e34bf4715: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer be6c68f5902a4725af474b575d30b50c at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42334: session id = be6c68f5902a4725af474b575d30b50c-00000000000000000000000000000000
I20260924 17:39:05.809793 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 17:39:05.810484 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 17:39:05.811895 6950 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: tablet copy: Starting download of 0 data blocks...
I20260924 17:39:05.812057 6950 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: tablet copy: Starting download of 1 WAL segments...
I20260924 17:39:05.813066 6950 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 17:39:05.814133 6506 tablet_copy_service.cc:342] P a57e962e8d03421597cf7b8e34bf4715: Request end of tablet copy session be6c68f5902a4725af474b575d30b50c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42334
I20260924 17:39:05.814205 6506 tablet_copy_service.cc:433] P a57e962e8d03421597cf7b8e34bf4715: ending tablet copy session be6c68f5902a4725af474b575d30b50c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer be6c68f5902a4725af474b575d30b50c
I20260924 17:39:05.816690 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:05.817930 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.818005 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:05.818044 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:05.818235 6950 server_base.cc:1034] running on GCE node
I20260924 17:39:05.818327 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:05.818462 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:05.819588 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790271545819586 us; error 40 us; skew 500 ppm
Sep 24 17:39:05 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:05.820916 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 17:39:05.821239 6950 webserver.cc:460] Webserver started at http://127.0.98.61:40073/ using document root <none> and password file <none>
I20260924 17:39:05.821367 6950 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:05.821412 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:05.822031 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:05.822582 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:05.822701 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:05.822749 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "be6c68f5902a4725af474b575d30b50c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "df988ac1c75d75ccdee7ab2a4573dcd2"
server_key_iv: "c163762bde9c9d650f1507e43d4d2b2f"
server_key_version: "encryptionkey@0"
I20260924 17:39:05.822808 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:05.842796 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:05.844494 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:05.847332 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33309
I20260924 17:39:05.847548 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33309 every 8 connection(s)
I20260924 17:39:05.847702 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
I20260924 17:39:05.848433 7121 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:39:05.848862 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Bootstrap starting.
I20260924 17:39:05.851195 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:39:05.851435 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Bootstrap complete.
I20260924 17:39:05.851640 7121 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } }
I20260924 17:39:05.851708 7121 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be6c68f5902a4725af474b575d30b50c, State: Initialized, Role: LEARNER
I20260924 17:39:05.851734 7121 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } }
I20260924 17:39:05.851933 7121 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:05.851938 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:05.852061 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: LEARNER
I20260924 17:39:05.853410 7135 catalog_manager.cc:1259] Loaded cluster ID: 1327fdf0142145b3988bf9b3650c3d55
I20260924 17:39:05.853797 7135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: loading cluster ID for follower catalog manager: success
I20260924 17:39:05.854578 7135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: acquiring CA information for follower catalog manager: success
I20260924 17:39:05.854866 7135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:39:05.902668 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:05.895756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.0.98.61:33309)
0924 17:39:05.895920 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:05.895922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:05.895929 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:39:05.895972 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:05.895973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:05.895982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:05.896064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:05.896654 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.897314 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.898100 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:05.898303 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:05.898410 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:05.899817 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:05.899819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:05.899819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:05.899829 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:05.901348 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.901753 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.901755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.901755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.901816 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:05.902076 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:05.902077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:05.902077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:05.902202 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:05.902312 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:05.902463 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 17:39:05.902557 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 17:39:05.929813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40275
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:06.013154 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:06.013422 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:06.013461 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:06.014788 7139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:06.014856 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:06.014878 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:06.014900 7139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:06.014932 7139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:06.016592 7139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:06.016881 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:06.017129 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:06.022377 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.022367 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.022488 7139 server_base.cc:1034] running on GCE node
W20260924 17:39:06.022418 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.022929 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:06.023340 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:06.024504 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790271546024480 us; error 48 us; skew 500 ppm
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:06.027266 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 17:39:06.027760 7139 webserver.cc:460] Webserver started at http://127.0.98.60:45379/ using document root <none> and password file <none>
I20260924 17:39:06.028005 7139 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.028052 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.028236 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:06.029609 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/instance:
uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
format_stamp: "Formatted at 2026-09-24 17:39:06 on dist-test-slave-d0pf"
server_key: "475ff07c3605b74800f9bb50a374106e"
server_key_iv: "cccaca281b18fad3c97eee5945c93854"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.030045 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal/instance:
uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
format_stamp: "Formatted at 2026-09-24 17:39:06 on dist-test-slave-d0pf"
server_key: "475ff07c3605b74800f9bb50a374106e"
server_key_iv: "cccaca281b18fad3c97eee5945c93854"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.031836 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 17:39:06.032604 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.032783 7139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:06.032886 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
format_stamp: "Formatted at 2026-09-24 17:39:06 on dist-test-slave-d0pf"
server_key: "475ff07c3605b74800f9bb50a374106e"
server_key_iv: "cccaca281b18fad3c97eee5945c93854"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.032989 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.045949 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:06.048467 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:06.048645 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:06.053483 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40275
I20260924 17:39:06.053540 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40275 every 8 connection(s)
I20260924 17:39:06.053937 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
I20260924 17:39:06.054930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7139
I20260924 17:39:06.055011 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:06.057451 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:06.066761 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.058272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.98.62:39945)
0924 17:39:06.058314 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.058316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.058406 (+ 90us) server_negotiation.cc:407] Connection header received
0924 17:39:06.058436 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.058436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.058447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.058513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.059251 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.059838 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.060809 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.061071 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.061126 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.063969 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.063971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.063971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.063981 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.065455 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.065846 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.065847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.065848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.065928 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.066160 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.066162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.066162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.066288 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.066364 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.066567 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.066650 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:39:06.066998 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.058137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42348)
0924 17:39:06.058288 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.058328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.058395 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.058576 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.058579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.059011 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.059123 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.059125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.059949 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.059950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.060722 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.060724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.061156 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.061157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.061239 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.061702 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.061713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.063808 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.065530 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.065537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.065541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.065773 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.065989 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.065990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.065991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.066088 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.066354 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.066360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.066436 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.066532 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.066758 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":71,"thread_start_us":96,"threads_started":1}
I20260924 17:39:06.067003 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.055665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.0.98.60:40275)
0924 17:39:06.055841 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.055849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.055877 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:39:06.056467 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.056474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.056530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.056657 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.057482 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.058718 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.059678 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.060249 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.060332 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.063479 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.063486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.063486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.063493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.065248 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.065692 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.065695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.065696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.065780 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.066027 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.066028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.066028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.066298 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.066472 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.066544 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.066689 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260924 17:39:06.067746 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:06.074621 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.068196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.98.61:33309)
0924 17:39:06.068241 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.068242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.068262 (+ 20us) server_negotiation.cc:407] Connection header received
0924 17:39:06.068293 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.068293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.068305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.068360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.068636 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.069134 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.069826 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.070040 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.070123 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.071993 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.071995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.071995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.072003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.073472 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.073828 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.073830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.073830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.073890 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.074090 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.074091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.074091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.074214 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.074289 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.074440 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.074527 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:39:06.074621 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.068113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56242)
0924 17:39:06.068146 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.068186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.068225 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.068364 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.068364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.068502 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.068552 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.068554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.069214 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.069215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.069730 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.069731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.070123 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.070123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.070192 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.070632 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.070637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.071836 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.073562 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.073563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.073564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.073737 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.073960 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.073960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.073961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.074026 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.074280 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.074281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.074325 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.074413 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.074521 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 17:39:06.075199 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 17:39:06.081990 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.075482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41294)
0924 17:39:06.075536 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.075546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.075592 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.075710 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.075711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.075843 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.075874 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.075875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.076553 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.076554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.077026 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.077027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.077420 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.077421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.077466 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.077893 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.077898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.079249 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.080900 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.080901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.080902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.081071 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.081279 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.081280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.081280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.081332 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.081604 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.081604 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.081643 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.081711 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.081815 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:39:06.081992 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.075539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.98.60:40275)
0924 17:39:06.075572 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.075574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.075582 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:06.075647 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.075648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.075658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.075700 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.075947 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.076487 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.077123 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.077344 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.077422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.079372 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.079373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.079373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.079380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.080784 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.081156 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.081157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.081158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.081214 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.081419 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.081419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.081420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.081540 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.081613 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.081736 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.081864 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":18}
I20260924 17:39:06.083632 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Bootstrap starting.
I20260924 17:39:06.084419 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:06.085098 7207 log.cc:824] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:06.085888 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: No bootstrap required, opened a new log
I20260924 17:39:06.087430 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } }
I20260924 17:39:06.087579 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:06.087603 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff6bf55e53e4b47afb763d68cd5eb8d, State: Initialized, Role: FOLLOWER
I20260924 17:39:06.087734 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } }
I20260924 17:39:06.088063 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } } }
I20260924 17:39:06.088141 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:06.088157 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:39:06.089830 7223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:39:06.089915 7223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:39:06.092526 7139 master_runner.cc:428] Detected that this master 3ff6bf55e53e4b47afb763d68cd5eb8d is joining an existing cluster
I20260924 17:39:06.092576 7139 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40275
I20260924 17:39:06.101078 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42362)
0924 17:39:06.093919 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.093930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.094018 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.094305 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.094306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.094502 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.094551 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.094552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.095399 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.095400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.096411 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.096414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.096928 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.096929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.096998 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.097529 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.097533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.097854 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.099698 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.099700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.099700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.100022 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.100317 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.100318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.100318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.100388 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.100705 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.100706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.100747 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.100807 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.100935 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":41,"thread_start_us":80,"threads_started":1}
I20260924 17:39:06.101441 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.0.98.62:39945)
0924 17:39:06.093692 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.093694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.093992 (+ 298us) server_negotiation.cc:407] Connection header received
0924 17:39:06.094220 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.094222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.094233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.094288 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.094652 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.095319 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.096505 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.096816 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.096896 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.097943 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.097944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.097944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.097953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.099507 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.100104 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.100106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.100107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.100173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.100471 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.100472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.100473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.100624 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.100682 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.101257 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.101341 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:39:06.106635 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.98.60:40275)
0924 17:39:06.093967 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.093969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.099728 (+ 5759us) server_negotiation.cc:407] Connection header received
0924 17:39:06.099820 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.099821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.099830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.099857 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.100625 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.101296 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.102220 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.102498 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.102543 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.103634 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.103635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.103636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.103642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.105467 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.105805 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.105806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.105806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.105862 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.106055 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.106056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.106056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.106189 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.106265 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.106422 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.106557 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":40}
I20260924 17:39:06.106635 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41298)
0924 17:39:06.099189 (+ 5396us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.099200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.099755 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.099925 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.099925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.100150 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.100546 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.100548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.101394 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.101395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.102136 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.102138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.102620 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.102620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.102686 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.103106 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.103111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.103533 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.105566 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.105567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.105567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.105727 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.105933 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.105934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.105934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.105988 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.106268 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.106270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.106308 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.106400 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.106510 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5378,"thread_start_us":79,"threads_started":1}
I20260924 17:39:06.115739 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56256)
0924 17:39:06.093966 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.093975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.094023 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.094172 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.094172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.094432 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.094521 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.094523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.095494 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.095495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.096411 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.096414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.097336 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.097337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.097427 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.111725 (+ 14298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.111731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.112154 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.114626 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.114627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.114627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.114832 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.115070 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.115070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.115071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.115129 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.115402 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.115402 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.115445 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.115505 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.115618 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":30,"thread_start_us":74,"threads_started":1}
I20260924 17:39:06.116307 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.093645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.98.61:33309)
0924 17:39:06.093692 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.093694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.094018 (+ 324us) server_negotiation.cc:407] Connection header received
0924 17:39:06.094074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.094075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.094086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.094159 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.094612 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.095417 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.096959 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.097256 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.097327 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.112259 (+ 14932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.112261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.112261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.112269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.114505 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.114904 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.114906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.114906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.114966 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.115197 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.115198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.115198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.115320 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.115378 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.116117 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.116198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 17:39:06.119196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6548
I20260924 17:39:06.121336 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.112385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.0.98.60:40275)
0924 17:39:06.112446 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.112448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.112475 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:39:06.112510 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.112510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.112521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.112576 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.114196 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.114927 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.115973 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.116249 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.116309 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.118291 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.118293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.118312 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.118321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.120193 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.120582 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.120583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.120583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.120633 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.120819 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.120820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.120820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.120949 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.121008 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.121167 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.121243 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:39:06.121490 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.112176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41308)
0924 17:39:06.112230 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.112242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.112295 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.112656 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.112657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.112847 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.112907 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.112911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.115015 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.115055 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.115881 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.115885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.116359 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.116360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.116440 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.116863 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.116868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.118193 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.120281 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.120282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.120282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.120506 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.120699 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.120700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.120700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.120760 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.121013 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.121014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.121059 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.121146 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.121247 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 17:39:06.122136 6485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40275: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40275
}
attrs {
promote: true
}
}
dest_uuid: "a57e962e8d03421597cf7b8e34bf4715"
cas_config_opid_index: 5
I20260924 17:39:06.122416 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
I20260924 17:39:06.122767 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.123781 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40275
I20260924 17:39:06.123862 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.123939 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.124141 7181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:06.124857 7181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 17:39:06.125177 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:06.125560 7139 master.cc:537] Master@127.0.98.60:40275 shutting down...
I20260924 17:39:06.126559 7181 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
W20260924 17:39:06.126647 7181 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
I20260924 17:39:06.127067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42723
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39945
--webserver_port=34591
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:39:06.127187 7181 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER be6c68f5902a4725af474b575d30b50c (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:06.127959 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57e962e8d03421597cf7b8e34bf4715. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:06.128051 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 17:39:06.127954 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:06.128088 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } }
I20260924 17:39:06.128140 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 17:39:06.128089 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 17:39:06.129397 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.129482 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.129820 7232 mvcc.cc:205] Tried to move back new op lower bound from 7332952248568938496 to 7332952248474935296. Current Snapshot: MvccSnapshot[applied={T|T < 7332952248484978688}]
W20260924 17:39:06.130196 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 -> Peer 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60:40275): Couldn't send request to peer 3ff6bf55e53e4b47afb763d68cd5eb8d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 17:39:06.131448 7139 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 17:39:06.131563 7139 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 17:39:06.131608 7139 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: stopping tablet replica
I20260924 17:39:06.133081 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.123214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56272)
0924 17:39:06.127148 (+ 3934us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.127162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.127225 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.129009 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.129010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.129014 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.129071 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.129073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.130076 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.130077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.131757 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.131761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.132744 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.132745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.132844 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.132885 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.132917 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.132974 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3913}
I20260924 17:39:06.133526 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.125275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.98.61:33309)
0924 17:39:06.128793 (+ 3518us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.128796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.128807 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:06.128836 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.128837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.128877 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.129137 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.129175 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.129961 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.131886 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.132636 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.133282 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.133368 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.133389 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.133435 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3473}
I20260924 17:39:06.133745 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 LEARNER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 17:39:06.133973 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 17:39:06.134608 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.136162 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57e962e8d03421597cf7b8e34bf4715. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.136272 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: LEARNER
I20260924 17:39:06.136318 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: true } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.136384 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: LEARNER
I20260924 17:39:06.136806 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: attempting to promote NON_VOTER be6c68f5902a4725af474b575d30b50c to VOTER
I20260924 17:39:06.137027 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
I20260924 17:39:06.137636 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.137484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41318)
0924 17:39:06.137568 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.137606 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 17:39:06.137789 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 LEARNER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 17:39:06.138113 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 17:39:06.138757 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.138669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41322)
0924 17:39:06.138705 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.138729 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 17:39:06.138942 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, be6c68f5902a4725af474b575d30b50c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.139848 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.139916 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.139971 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.140010 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.140367 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, be6c68f5902a4725af474b575d30b50c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.141281 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.141388 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:39:06.141460 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 -> Peer 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60:40275): Couldn't send request to peer 3ff6bf55e53e4b47afb763d68cd5eb8d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 17:39:06.147732 7139 master.cc:552] Master@127.0.98.60:40275 shutdown complete.
I20260924 17:39:06.147787 7139 master_runner.cc:305] Clearing existing system tablet
I20260924 17:39:06.147931 7139 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.147972 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.148756 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:06.149353 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.149501 7139 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.149546 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.149569 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
format_stamp: "Formatted at 2026-09-24 17:39:06 on dist-test-slave-d0pf"
server_key: "475ff07c3605b74800f9bb50a374106e"
server_key_iv: "cccaca281b18fad3c97eee5945c93854"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.150321 7139 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:06.151835 7139 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 17:39:06.151925 7139 log.cc:1196] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 17:39:06.152189 7139 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Deleting consensus metadata
I20260924 17:39:06.152407 7139 master_runner.cc:315] Copying system tablet from 127.0.98.62:39945
I20260924 17:39:06.153131 7139 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39945
I20260924 17:39:06.162614 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.153603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.0.98.62:39945)
0924 17:39:06.153653 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.153656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.153937 (+ 281us) server_negotiation.cc:407] Connection header received
0924 17:39:06.154048 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.154049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.154062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.154095 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.154884 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.156110 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.157421 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.157795 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.157869 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.159341 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.159343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.159343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.159352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.161240 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.161686 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.161687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.161688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.161745 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.161963 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.161964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.161965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.162116 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.162161 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.162374 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.162469 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:39:06.163213 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.153692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42364)
0924 17:39:06.153818 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.153828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.153966 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.154167 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.154168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.154407 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.154761 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.154764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.156230 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.156231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.157305 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.157308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.157963 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.157964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.158048 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.158636 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.158642 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.159165 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.161342 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.161344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.161344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.161599 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.161818 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.161819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.161819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.161890 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.162207 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.162208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.162259 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.162781 (+ 522us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.162922 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 17:39:06.163462 6506 tablet_copy_service.cc:140] P a57e962e8d03421597cf7b8e34bf4715: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ff6bf55e53e4b47afb763d68cd5eb8d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42364)
I20260924 17:39:06.163533 6506 tablet_copy_service.cc:161] P a57e962e8d03421597cf7b8e34bf4715: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ff6bf55e53e4b47afb763d68cd5eb8d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42364: session id = 3ff6bf55e53e4b47afb763d68cd5eb8d-00000000000000000000000000000000
I20260924 17:39:06.164332 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 17:39:06.164897 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 17:39:06.166428 7139 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: tablet copy: Starting download of 0 data blocks...
I20260924 17:39:06.166606 7139 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: tablet copy: Starting download of 1 WAL segments...
I20260924 17:39:06.167313 7139 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 17:39:06.168323 6506 tablet_copy_service.cc:342] P a57e962e8d03421597cf7b8e34bf4715: Request end of tablet copy session 3ff6bf55e53e4b47afb763d68cd5eb8d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 17:39:06.168385 6506 tablet_copy_service.cc:433] P a57e962e8d03421597cf7b8e34bf4715: ending tablet copy session 3ff6bf55e53e4b47afb763d68cd5eb8d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ff6bf55e53e4b47afb763d68cd5eb8d
I20260924 17:39:06.171592 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:06.172887 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.173427 7139 server_base.cc:1034] running on GCE node
W20260924 17:39:06.176095 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.176124 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.176370 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:06.176537 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:06.177670 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790271546177661 us; error 30 us; skew 500 ppm
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:06.179458 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 17:39:06.179744 7139 webserver.cc:460] Webserver started at http://127.0.98.60:44947/ using document root <none> and password file <none>
I20260924 17:39:06.179881 7139 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.179914 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.187327 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260924 17:39:06.188095 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.188259 7139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:39:06.188302 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
format_stamp: "Formatted at 2026-09-24 17:39:06 on dist-test-slave-d0pf"
server_key: "475ff07c3605b74800f9bb50a374106e"
server_key_iv: "cccaca281b18fad3c97eee5945c93854"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.188372 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.211869 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:06.213778 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:06.216938 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40275
I20260924 17:39:06.216967 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40275 every 8 connection(s)
I20260924 17:39:06.217285 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
I20260924 17:39:06.218108 7313 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:39:06.218652 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Bootstrap starting.
I20260924 17:39:06.220521 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:39:06.220760 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: Bootstrap complete.
I20260924 17:39:06.220958 7313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
I20260924 17:39:06.221046 7313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ff6bf55e53e4b47afb763d68cd5eb8d, State: Initialized, Role: LEARNER
I20260924 17:39:06.221091 7313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } }
I20260924 17:39:06.221285 7313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:06.221257 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.221464 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: LEARNER
I20260924 17:39:06.222739 7327 catalog_manager.cc:1259] Loaded cluster ID: 1327fdf0142145b3988bf9b3650c3d55
I20260924 17:39:06.222823 7327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: loading cluster ID for follower catalog manager: success
I20260924 17:39:06.223884 7327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: acquiring CA information for follower catalog manager: success
I20260924 17:39:06.224172 7327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 17:39:06.263060 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:06.263374 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:06.263418 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:06.265333 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:06.265408 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:06.265452 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:06.267839 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34591
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:06.268146 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:06.268438 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:06.274386 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.274432 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.274483 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.274881 7236 server_base.cc:1034] running on GCE node
I20260924 17:39:06.275141 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:06.275417 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:06.276595 7236 hybrid_clock.cc:648] HybridClock initialized: now 1790271546276574 us; error 38 us; skew 500 ppm
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:06.279287 7236 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 17:39:06.279735 7236 webserver.cc:460] Webserver started at http://127.0.98.1:34591/ using document root <none> and password file <none>
I20260924 17:39:06.279978 7236 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.280064 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.281504 7236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:06.282213 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.282456 7236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:06.282562 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "7ecb344160a643a39efa254fc861e47c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "22480c1e7777799540c58fa310ede455"
server_key_iv: "c1e5bf21347ef2ae0a46f4eb87c4cdb8"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.282950 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.293989 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:06.296505 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:06.296679 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:06.296918 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:06.297240 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:06.297299 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.297343 7236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:06.297367 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.304065 7236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42723
I20260924 17:39:06.304107 7455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42723 every 8 connection(s)
I20260924 17:39:06.304466 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:06.313659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7236
I20260924 17:39:06.313822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6682
I20260924 17:39:06.316205 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.305207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45219 (local address 127.0.98.62:39945)
0924 17:39:06.305254 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.305257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.305512 (+ 255us) server_negotiation.cc:407] Connection header received
0924 17:39:06.306204 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.306205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.306219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.306249 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.307356 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.308106 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309170 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309460 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309499 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.312410 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.312412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.312412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.312420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.314141 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315076 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315140 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315456 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315602 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.315647 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.316009 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.316100 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:39:06.316341 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.306461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51003 (local address 127.0.98.61:33309)
0924 17:39:06.306509 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.306511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.306621 (+ 110us) server_negotiation.cc:407] Connection header received
0924 17:39:06.306713 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.306714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.306724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.306779 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.307344 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.307889 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309010 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309242 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309318 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.313080 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.313082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.313083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.313090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.315164 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315529 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315580 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315777 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315912 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.315980 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.316173 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.316245 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:39:06.316445 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.305548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.98.1:45737)
0924 17:39:06.305678 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.305687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.306087 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.306394 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.306398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.306905 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.307242 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.307248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.308417 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.308418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.309190 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.309191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309620 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.309698 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.310296 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.310301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.311831 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.314652 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.314653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.314654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.314958 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.315187 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.315189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.315340 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.315652 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.315657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.315842 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.315956 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.316153 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":31,"thread_start_us":64,"threads_started":1}
I20260924 17:39:06.316622 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.305236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:45219)
0924 17:39:06.305390 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.305426 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.306078 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.306394 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.306398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.306905 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.307242 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.307248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.308195 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.308196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.309078 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.309080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309576 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.309660 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.310093 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.310104 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.312303 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.314554 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.314564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.314567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.314944 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.315216 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.315217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.315337 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.315692 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.315693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.315836 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.316325 (+ 489us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.316501 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":14,"thread_start_us":90,"threads_started":1}
I20260924 17:39:06.316726 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.306402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.98.1:51003)
0924 17:39:06.306561 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.306572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.306647 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.306782 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.306782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.307002 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.307242 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.307248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.308002 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.308003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.308898 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.308900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309322 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.309453 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.310463 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.310469 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.312911 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.315254 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.315255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.315448 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.315654 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.315655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.315709 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.316028 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.316029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.316066 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.316417 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.316553 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":43,"thread_start_us":78,"threads_started":1}
I20260924 17:39:06.316974 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.305710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45737 (local address 127.0.98.60:40275)
0924 17:39:06.305821 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.305823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.305831 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:06.306203 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.306205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.306220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.306309 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.307350 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.308335 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309265 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.309533 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.309618 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.311951 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.311952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.311952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.311958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.314567 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315059 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315120 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.315409 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.315410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.315410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.315568 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.315627 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.316712 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.316814 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260924 17:39:06.317518 7458 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33309
I20260924 17:39:06.317629 7458 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.317641 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40275
I20260924 17:39:06.317691 7456 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.317747 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:06.317778 7457 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.317860 7456 heartbeater.cc:511] Master 127.0.98.60:40275 requested a full tablet report, sending...
I20260924 17:39:06.317857 7457 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:06.317857 7458 heartbeater.cc:511] Master 127.0.98.61:33309 requested a full tablet report, sending...
I20260924 17:39:06.318298 6485 ts_manager.cc:194] Re-registered known tserver with Master: 7ecb344160a643a39efa254fc861e47c (127.0.98.1:42723)
I20260924 17:39:06.318403 7277 ts_manager.cc:194] Registered new tserver with Master: 7ecb344160a643a39efa254fc861e47c (127.0.98.1:42723)
I20260924 17:39:06.318677 7085 ts_manager.cc:194] Registered new tserver with Master: 7ecb344160a643a39efa254fc861e47c (127.0.98.1:42723)
I20260924 17:39:06.318758 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45219
I20260924 17:39:06.321666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36183
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39945
--webserver_port=39583
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:06.404050 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:06.404264 7464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:06.404289 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:06.405495 7464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:06.405542 7464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:06.405571 7464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:06.407130 7464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39583
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:06.407420 7464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:06.407662 7464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:06.412842 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.412842 7472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.413115 7464 server_base.cc:1034] running on GCE node
W20260924 17:39:06.412875 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.413528 7464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:06.413805 7464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:06.414988 7464 hybrid_clock.cc:648] HybridClock initialized: now 1790271546414966 us; error 33 us; skew 500 ppm
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:06.417541 7464 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 17:39:06.417950 7464 webserver.cc:460] Webserver started at http://127.0.98.2:39583/ using document root <none> and password file <none>
I20260924 17:39:06.418143 7464 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.418193 7464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.419602 7464 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:06.420171 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.420362 7464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:39:06.420450 7464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "46f41884be83c3c8929bbe3021cba9e8"
server_key_iv: "78a17a9a05f7c73f8c4c36f9072c6770"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.420794 7464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.430392 7464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:06.433099 7464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:06.433223 7464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:06.433416 7464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:06.433687 7464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:06.433718 7464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.433743 7464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:06.433754 7464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.440016 7464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36183
I20260924 17:39:06.440130 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36183 every 8 connection(s)
I20260924 17:39:06.440465 7464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:06.446635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7464
I20260924 17:39:06.446820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6816
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 17:39:06.452862 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41831 (local address 127.0.98.62:39945)
0924 17:39:06.441746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.441748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.442427 (+ 679us) server_negotiation.cc:407] Connection header received
0924 17:39:06.442527 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.442528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.442539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.442608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.443340 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.443856 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.444947 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445148 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445234 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.449151 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.449153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.449153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.449162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.450917 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451650 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.451651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.451651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.451723 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.452080 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.452081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.452082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.452220 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.452276 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.452676 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.452761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:06.452862 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57391 (local address 127.0.98.60:40275)
0924 17:39:06.441410 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.441413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.441662 (+ 249us) server_negotiation.cc:407] Connection header received
0924 17:39:06.442315 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.442316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.442330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.442401 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.443376 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.444139 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445327 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445653 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445743 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.448556 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.448558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.448558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.448566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.450849 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451649 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.451650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.451650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.451792 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.452119 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.452120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.452120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.452243 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.452293 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.452666 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.452763 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:39:06.452925 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48653 (local address 127.0.98.61:33309)
0924 17:39:06.441414 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.441416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.441813 (+ 397us) server_negotiation.cc:407] Connection header received
0924 17:39:06.442315 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.442316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.442330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.442401 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.443364 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.444139 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445302 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445517 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445551 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.449262 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.449263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.449264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.449274 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.451123 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451707 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.451709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.451709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.451767 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.452061 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.452062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.452062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.452230 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.452275 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.452663 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.452777 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 17:39:06.453473 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:41831)
0924 17:39:06.442368 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.442377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.442455 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.442634 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.442635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.442967 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.443247 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.443253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.443970 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.443971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.444848 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.444851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445242 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.445386 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.446026 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.446035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.448973 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.450995 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.450996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.450996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451552 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.451901 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.451902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451999 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.452315 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.452323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.452501 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.452987 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.453216 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":34,"thread_start_us":58,"threads_started":1}
I20260924 17:39:06.453481 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.98.2:57391)
0924 17:39:06.441505 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.441563 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.442185 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.442549 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.442550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.442982 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.443247 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.443253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.444242 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.444243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.445198 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.445201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445745 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.445821 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.446342 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.446347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.448428 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.450953 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.450963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.450965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451552 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.451888 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.451890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451996 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.452325 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.452325 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.452501 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.452987 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.453260 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":131,"thread_start_us":89,"threads_started":1}
I20260924 17:39:06.454507 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.441292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.98.2:48653)
0924 17:39:06.441728 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.441736 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.442194 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.442487 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.442492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.442962 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.443247 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.443253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.444242 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.444243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.445201 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.445203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.445648 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.445649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.445714 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.446147 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.446152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.448955 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.451214 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.451216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451619 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.451833 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.451834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.451834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.451989 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.452335 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.452336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.452502 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.454047 (+ 1545us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.454175 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":47,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260924 17:39:06.454883 7595 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33309
I20260924 17:39:06.455005 7595 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.455248 7595 heartbeater.cc:511] Master 127.0.98.61:33309 requested a full tablet report, sending...
I20260924 17:39:06.455638 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40275
I20260924 17:39:06.455647 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:06.455700 7593 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.455706 7594 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.455780 7593 heartbeater.cc:511] Master 127.0.98.60:40275 requested a full tablet report, sending...
I20260924 17:39:06.455780 7594 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:06.455986 7085 ts_manager.cc:194] Registered new tserver with Master: 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183)
I20260924 17:39:06.456275 6485 ts_manager.cc:194] Re-registered known tserver with Master: 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183)
I20260924 17:39:06.456322 7277 ts_manager.cc:194] Registered new tserver with Master: 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183)
I20260924 17:39:06.456823 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41831
I20260924 17:39:06.456830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45857
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39945
--webserver_port=36639
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:06.542492 7600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:06.542716 7600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:06.542740 7600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:06.544024 7600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:06.544076 7600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:06.544106 7600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:06.545789 7600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36639
--tserver_master_addrs=127.0.98.62:39945,127.0.98.61:33309,127.0.98.60:40275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:06.546048 7600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:06.546319 7600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:06.551533 7606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:06.551518 7605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.551626 7600 server_base.cc:1034] running on GCE node
W20260924 17:39:06.551517 7608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:06.551956 7600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:06.552254 7600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:06.553476 7600 hybrid_clock.cc:648] HybridClock initialized: now 1790271546553458 us; error 45 us; skew 500 ppm
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:06.556192 7600 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 17:39:06.556692 7600 webserver.cc:460] Webserver started at http://127.0.98.3:36639/ using document root <none> and password file <none>
I20260924 17:39:06.556945 7600 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:06.557029 7600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:06.558526 7600 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:39:06.559350 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.559544 7600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:06.559651 7600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "3e63bb4cad2449489560ab8e63f4e588"
format_stamp: "Formatted at 2026-09-24 17:39:05 on dist-test-slave-d0pf"
server_key: "1397c291cc2fdf67f2be26a7796d7387"
server_key_iv: "c320fc201700eb764cbd3b81eb09ce04"
server_key_version: "encryptionkey@0"
I20260924 17:39:06.560026 7600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:06.588609 7600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:06.591032 7600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:06.591244 7600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:06.591501 7600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:06.591835 7600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:06.591892 7600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.591935 7600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:06.591965 7600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:06.598551 7600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45857
I20260924 17:39:06.598629 7728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45857 every 8 connection(s)
I20260924 17:39:06.599001 7600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:06.600502 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.597264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41328)
0924 17:39:06.597355 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.597368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.597426 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.597581 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.597582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.597583 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.597637 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.597640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.598343 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.598345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.599342 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.599347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.600158 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.600159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.600239 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.600244 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.600317 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.600380 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 17:39:06.600750 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.597311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.0.98.60:40275)
0924 17:39:06.597360 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.597362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.597407 (+ 45us) server_negotiation.cc:407] Connection header received
0924 17:39:06.597481 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.597482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.597507 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.597576 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.597698 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.598240 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.599477 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.600005 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.600079 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.600129 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.600612 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.600669 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:06.601200 7287 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 17:39:06.601378 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57e962e8d03421597cf7b8e34bf4715. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.601538 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: LEARNER
I20260924 17:39:06.602260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7600
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 17:39:06 dist-test-slave-d0pf krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:06.613361 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.600390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.98.3:41275)
0924 17:39:06.601113 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.601125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.601748 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.602077 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.602078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.602501 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.602840 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.602848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603695 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.603696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.604523 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.604525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.605084 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.605257 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.605925 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.605938 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.609089 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.611277 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.611290 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.611292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.611690 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.611949 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.611949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.611950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.612121 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.612417 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.612425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.612655 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.612826 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.613041 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":36,"thread_start_us":75,"threads_started":1}
I20260924 17:39:06.613704 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.600576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41275 (local address 127.0.98.61:33309)
0924 17:39:06.600616 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.600618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.601174 (+ 556us) server_negotiation.cc:407] Connection header received
0924 17:39:06.601960 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.601962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.601973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.602047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.602969 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603563 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.604652 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.604942 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605014 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.609196 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.609197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.609198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.609206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.611183 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.611811 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.611813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.611814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.611873 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.612200 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.612201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.612201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.612354 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.612421 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.613532 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.613616 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:39:06.614349 7731 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33309
I20260924 17:39:06.614470 7731 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.614533 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.599912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45439 (local address 127.0.98.62:39945)
0924 17:39:06.599986 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.599988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.600379 (+ 391us) server_negotiation.cc:407] Connection header received
0924 17:39:06.601838 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.601839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.601851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.601896 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.602954 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603838 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605307 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.606073 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.606427 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.610217 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.610219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.610219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.610227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.612173 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.613010 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.613012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.613012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.613071 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.613275 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.613276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.613276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.613401 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.614105 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.614338 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.614421 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:39:06.614697 7731 heartbeater.cc:511] Master 127.0.98.61:33309 requested a full tablet report, sending...
I20260924 17:39:06.614732 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.600554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48099 (local address 127.0.98.60:40275)
0924 17:39:06.600837 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.600839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.600963 (+ 124us) server_negotiation.cc:407] Connection header received
0924 17:39:06.602099 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.602100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.602110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.602155 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.602953 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603838 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605315 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.605634 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605684 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.608860 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.608862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.608862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.608870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.612170 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.612950 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.612951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.612951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.613012 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.613220 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.613221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.613221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.613354 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.613396 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.614317 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.614646 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
I20260924 17:39:06.614902 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.599924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:45439)
0924 17:39:06.600254 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.600303 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.601744 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.601976 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.601981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.602508 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.602839 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.602845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603955 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.603956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.605176 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.605179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.606524 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.606525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.606608 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.607348 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.607354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.610087 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.612472 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.612473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.612473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.612914 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.613144 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.613144 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.613144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.613202 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.614157 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.614157 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.614215 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.614662 (+ 447us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.614789 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":176,"thread_start_us":216,"threads_started":1}
I20260924 17:39:06.615108 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.600388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.98.3:48099)
0924 17:39:06.600898 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.600908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.601752 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.602196 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.602197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.602502 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:06.602839 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.602845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.603955 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.603956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.605214 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.605216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.605841 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.605841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.605935 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.606409 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:06.606414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:06.608731 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:06.612309 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.612309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.612310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.612862 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.613090 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:06.613091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:06.613091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:06.613149 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:06.613447 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:06.613448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:06.613510 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.614836 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.614928 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":93,"thread_start_us":69,"threads_started":1}
I20260924 17:39:06.615195 7085 ts_manager.cc:194] Registered new tserver with Master: 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:06.615510 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260924 17:39:06.615571 7730 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.615651 7730 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260924 17:39:06.616257 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40275
I20260924 17:39:06.616313 7729 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:06.616331 6485 ts_manager.cc:194] Re-registered known tserver with Master: 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:06.616375 7729 heartbeater.cc:511] Master 127.0.98.60:40275 requested a full tablet report, sending...
I20260924 17:39:06.616667 7277 ts_manager.cc:194] Registered new tserver with Master: 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:06.616797 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45439
I20260924 17:39:06.618124 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.605762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.0.98.60:40275)
0924 17:39:06.606200 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.606202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.606226 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:39:06.606254 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.606255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.606269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.606334 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.607884 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.608687 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.609824 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.610134 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.610178 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.614145 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.614146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.614148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.614157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.616802 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.617221 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.617222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.617222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.617289 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.617494 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.617495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.617495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.617648 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.617724 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.617947 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.618031 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260924 17:39:06.619438 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.604855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.0.98.62:39945)
0924 17:39:06.606432 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.606434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.606443 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:06.606466 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.606467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.606483 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.606560 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.607432 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.608566 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.609818 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.610153 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.610208 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.612080 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.612082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.612082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.612091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.617576 (+ 5485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.617935 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.617936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.617937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.618002 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.618493 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.618495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.618496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.618661 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.618775 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.618990 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.619228 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546,"thread_start_us":81,"threads_started":1}
I20260924 17:39:06.621609 6485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:06.624607 6485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:06.624959 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.605267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.61:33309)
0924 17:39:06.605805 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.605807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.612659 (+ 6852us) server_negotiation.cc:407] Connection header received
0924 17:39:06.615239 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.615241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.615256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.615311 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.616021 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.617137 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.618458 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.618773 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.618815 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.621346 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.621347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.621347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.621358 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.623617 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.624011 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.624013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.624013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.624083 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.624324 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.624325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.624326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.624487 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.624585 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.624795 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.624875 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":57,"threads_started":1}
I20260924 17:39:06.625710 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: true } } }
I20260924 17:39:06.625826 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.626609 7045 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: attempting to promote NON_VOTER 3ff6bf55e53e4b47afb763d68cd5eb8d to VOTER
I20260924 17:39:06.626758 7045 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } }
I20260924 17:39:06.627656 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 17:39:06.627677 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 LEARNER]: Refusing update from remote peer a57e962e8d03421597cf7b8e34bf4715: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 17:39:06.627966 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 17:39:06.628317 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 17:39:06.629066 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.629238 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.629402 7287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630059 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630136 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.630115 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630203 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.630154 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630189 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630234 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:39:06.630252 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:06.630451 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a57e962e8d03421597cf7b8e34bf4715" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:06.630563 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:39:06.636660 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.632830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45857 (local address 127.0.0.1:49930)
0924 17:39:06.633183 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.633193 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.633240 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.633424 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.633425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.633427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.633485 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.633487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634338 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.634339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.635445 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.635448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636294 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.636295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.636356 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.636360 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.636418 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.636463 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":68,"threads_started":1}
I20260924 17:39:06.636672 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.633025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.0.98.3:45857)
0924 17:39:06.633259 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.633264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.633275 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:06.633314 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.633316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.633330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.633430 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.633546 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634225 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.635553 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636192 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.636294 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.636382 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.636437 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.636481 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260924 17:39:06.636952 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.632608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42723 (local address 127.0.0.1:51626)
0924 17:39:06.632872 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.632883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.632934 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.633269 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.633270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.633273 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.633335 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.633337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634704 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.634705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.635625 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.635627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636544 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.636545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.636712 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.636718 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.636765 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.636812 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":22,"thread_start_us":97,"threads_started":1}
I20260924 17:39:06.637209 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.632594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36183 (local address 127.0.0.1:38412)
0924 17:39:06.632692 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.632705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.632749 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.633327 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.633328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.633329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.633359 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.633360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634680 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.634681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.635752 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.635754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636930 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.636931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.637031 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.637034 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.637088 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.637126 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 17:39:06.637230 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.632762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.0.98.2:36183)
0924 17:39:06.632928 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.632934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.632945 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:06.632992 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.632994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.633011 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.633433 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.633490 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634595 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.635832 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636809 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.636922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.637002 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.637100 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.637145 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260924 17:39:06.637413 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.632695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.98.1:42723)
0924 17:39:06.633094 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.633099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.633108 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:06.633144 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.633145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.633161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.633412 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.633473 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.634582 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.635715 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.636438 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.637070 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.637152 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.637176 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.637340 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":75,"threads_started":1}
I20260924 17:39:06.638023 7527 tablet_service.cc:1467] Processing CreateTablet for tablet ec5349036eff40bcac6ae4f570c025e5 (DEFAULT_TABLE table=test-table [id=ef82e507850b4c50a6a1b77e123e4dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:06.638154 7390 tablet_service.cc:1467] Processing CreateTablet for tablet ec5349036eff40bcac6ae4f570c025e5 (DEFAULT_TABLE table=test-table [id=ef82e507850b4c50a6a1b77e123e4dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:06.638159 7663 tablet_service.cc:1467] Processing CreateTablet for tablet ec5349036eff40bcac6ae4f570c025e5 (DEFAULT_TABLE table=test-table [id=ef82e507850b4c50a6a1b77e123e4dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:06.638439 7527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5349036eff40bcac6ae4f570c025e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:06.638454 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5349036eff40bcac6ae4f570c025e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:06.638572 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5349036eff40bcac6ae4f570c025e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:06.641986 7758 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Bootstrap starting.
I20260924 17:39:06.642513 7760 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Bootstrap starting.
I20260924 17:39:06.642802 7758 tablet_bootstrap.cc:654] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:06.643174 7760 tablet_bootstrap.cc:654] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:06.643188 7758 log.cc:824] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:06.643201 7759 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Bootstrap starting.
I20260924 17:39:06.643666 7760 log.cc:824] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:06.643954 7758 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: No bootstrap required, opened a new log
I20260924 17:39:06.643968 7759 tablet_bootstrap.cc:654] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:06.644035 7758 ts_tablet_manager.cc:1397] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:06.644464 7759 log.cc:824] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:06.644620 7760 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: No bootstrap required, opened a new log
I20260924 17:39:06.644695 7760 ts_tablet_manager.cc:1397] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:39:06.645465 7759 tablet_bootstrap.cc:492] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: No bootstrap required, opened a new log
I20260924 17:39:06.645566 7759 ts_tablet_manager.cc:1397] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:39:06.646283 7760 raft_consensus.cc:348] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.646472 7760 raft_consensus.cc:374] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:06.646525 7760 raft_consensus.cc:729] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecb344160a643a39efa254fc861e47c, State: Initialized, Role: FOLLOWER
I20260924 17:39:06.646811 7760 consensus_queue.cc:260] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.647174 7760 ts_tablet_manager.cc:1428] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 17:39:06.647199 7758 raft_consensus.cc:348] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.647367 7758 raft_consensus.cc:374] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:06.647413 7758 raft_consensus.cc:729] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e63bb4cad2449489560ab8e63f4e588, State: Initialized, Role: FOLLOWER
I20260924 17:39:06.647562 7758 consensus_queue.cc:260] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.647733 7457 heartbeater.cc:503] Master 127.0.98.62:39945 was elected leader, sending a full tablet report...
I20260924 17:39:06.647871 7758 ts_tablet_manager.cc:1428] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:39:06.648233 7730 heartbeater.cc:503] Master 127.0.98.62:39945 was elected leader, sending a full tablet report...
I20260924 17:39:06.648883 7759 raft_consensus.cc:348] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.649115 7759 raft_consensus.cc:374] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:06.649159 7759 raft_consensus.cc:729] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d98c45d3ab430bb8e1962ce54a1a0c, State: Initialized, Role: FOLLOWER
I20260924 17:39:06.649271 7759 consensus_queue.cc:260] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.649583 7759 ts_tablet_manager.cc:1428] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:39:06.649616 7594 heartbeater.cc:503] Master 127.0.98.62:39945 was elected leader, sending a full tablet report...
I20260924 17:39:06.681219 7764 raft_consensus.cc:482] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:06.681349 7764 raft_consensus.cc:504] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.681959 7764 leader_election.cc:288] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183), 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:06.685217 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.682242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45857 (local address 127.0.98.1:36053)
0924 17:39:06.682308 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.682320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.682369 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.682534 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.682535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.682538 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.682619 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.682621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.683513 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.683514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.684314 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.684317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.684885 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.684984 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.684987 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.685058 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.685110 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260924 17:39:06.685361 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.682355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36053 (local address 127.0.98.3:45857)
0924 17:39:06.682404 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.682406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.682417 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:06.682444 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.682444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.682458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.682509 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.682729 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.683427 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684426 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.684797 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684885 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.684896 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.685237 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.685292 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 17:39:06.685384 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.682153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36183 (local address 127.0.98.1:36821)
0924 17:39:06.682216 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:06.682228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:06.682273 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:06.682497 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:06.682498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:06.682500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:06.682606 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.682609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.683581 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.683582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.684318 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:06.684320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.684891 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:06.684972 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.684975 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:06.685264 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 17:39:06.685307 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260924 17:39:06.685407 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.682249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36821 (local address 127.0.98.2:36183)
0924 17:39:06.682355 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.682357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.682365 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:06.682399 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.682399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.682413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.682467 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:06.682727 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.683487 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684423 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.684797 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.684880 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.684893 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.685277 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.685324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 17:39:06.685722 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5349036eff40bcac6ae4f570c025e5" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" is_pre_election: true
I20260924 17:39:06.685696 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5349036eff40bcac6ae4f570c025e5" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e63bb4cad2449489560ab8e63f4e588" is_pre_election: true
I20260924 17:39:06.685864 7547 raft_consensus.cc:2454] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 0.
I20260924 17:39:06.685864 7683 raft_consensus.cc:2454] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 0.
I20260924 17:39:06.686113 7343 leader_election.cc:302] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e63bb4cad2449489560ab8e63f4e588, 7ecb344160a643a39efa254fc861e47c; no voters:
I20260924 17:39:06.686384 7764 raft_consensus.cc:2781] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:06.686467 7764 raft_consensus.cc:482] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:06.686504 7764 raft_consensus.cc:3037] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:06.687361 7764 raft_consensus.cc:504] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.687529 7764 leader_election.cc:288] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 election: Requested vote from peers 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183), 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:06.687664 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5349036eff40bcac6ae4f570c025e5" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
I20260924 17:39:06.687745 7547 raft_consensus.cc:3037] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:06.687731 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5349036eff40bcac6ae4f570c025e5" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e63bb4cad2449489560ab8e63f4e588"
I20260924 17:39:06.687799 7683 raft_consensus.cc:3037] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:06.688506 7547 raft_consensus.cc:2454] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 1.
I20260924 17:39:06.688722 7683 raft_consensus.cc:2454] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 1.
I20260924 17:39:06.688723 7346 leader_election.cc:302] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d98c45d3ab430bb8e1962ce54a1a0c, 7ecb344160a643a39efa254fc861e47c; no voters:
I20260924 17:39:06.688835 7764 raft_consensus.cc:2781] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:06.689054 7764 raft_consensus.cc:686] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 1 LEADER]: Becoming Leader. State: Replica: 7ecb344160a643a39efa254fc861e47c, State: Running, Role: LEADER
I20260924 17:39:06.689172 7764 consensus_queue.cc:237] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:06.690063 6486 catalog_manager.cc:5495] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ecb344160a643a39efa254fc861e47c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7ecb344160a643a39efa254fc861e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } health_report { overall_health: UNKNOWN } } }
W20260924 17:39:06.691213 7596 tablet.cc:2369] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:06.699554 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.696962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.0.98.1:42723)
0924 17:39:06.697002 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.697003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.697009 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:39:06.697040 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.697040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.697059 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.697112 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:06.697304 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.697775 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.698563 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.698779 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.698855 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.699120 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:06.699242 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:06.699329 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.699420 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.699442 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:39:06.701632 7547 raft_consensus.cc:1261] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Refusing update from remote peer 7ecb344160a643a39efa254fc861e47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:06.701648 7683 raft_consensus.cc:1261] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Refusing update from remote peer 7ecb344160a643a39efa254fc861e47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:06.702032 7764 consensus_queue.cc:1035] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:06.702209 7767 consensus_queue.cc:1035] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:06.705998 7770 mvcc.cc:205] Tried to move back new op lower bound from 7332952255286632448 to 7332952255239331840. Current Snapshot: MvccSnapshot[applied={T|T < 7332952255286632448}]
I20260924 17:39:06.706503 7772 mvcc.cc:205] Tried to move back new op lower bound from 7332952255286632448 to 7332952255239331840. Current Snapshot: MvccSnapshot[applied={T|T < 7332952255286632448 or (T in {7332952255286632448})}]
I20260924 17:39:06.709149 6485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:06.709273 6485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:06.711151 6485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a57e962e8d03421597cf7b8e34bf4715: Sending DeleteTablet for 3 replicas of tablet ec5349036eff40bcac6ae4f570c025e5
I20260924 17:39:06.711517 7390 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5349036eff40bcac6ae4f570c025e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51626
I20260924 17:39:06.711503 7527 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5349036eff40bcac6ae4f570c025e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38412
I20260924 17:39:06.711649 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5349036eff40bcac6ae4f570c025e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49930
I20260924 17:39:06.711941 7781 tablet_replica.cc:330] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: stopping tablet replica
I20260924 17:39:06.712237 7781 raft_consensus.cc:2229] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:06.712393 7781 raft_consensus.cc:2258] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:06.712814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6454
I20260924 17:39:06.712984 7781 ts_tablet_manager.cc:1905] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:06.719305 7781 ts_tablet_manager.cc:1918] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:06.719635 7781 log.cc:1196] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/ec5349036eff40bcac6ae4f570c025e5
I20260924 17:39:06.720095 7781 ts_tablet_manager.cc:1939] T ec5349036eff40bcac6ae4f570c025e5 P 3e63bb4cad2449489560ab8e63f4e588: Deleting consensus metadata
I20260924 17:39:06.722729 7780 tablet_replica.cc:330] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: stopping tablet replica
I20260924 17:39:06.722894 7780 raft_consensus.cc:2229] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:06.723100 7780 raft_consensus.cc:2258] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:06.723677 7780 ts_tablet_manager.cc:1905] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:06.725436 7780 ts_tablet_manager.cc:1918] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:06.725519 7780 log.cc:1196] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/ec5349036eff40bcac6ae4f570c025e5
I20260924 17:39:06.725790 7780 ts_tablet_manager.cc:1939] T ec5349036eff40bcac6ae4f570c025e5 P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting consensus metadata
W20260924 17:39:06.726195 7483 connection.cc:441] server connection from 127.0.0.1:38412 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38412 (request call id 1) could send its response
I20260924 17:39:06.726380 7779 tablet_replica.cc:330] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: stopping tablet replica
I20260924 17:39:06.726523 7779 raft_consensus.cc:2229] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:06.726670 7779 raft_consensus.cc:2258] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:06.731999 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.723650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.98.60:40275)
0924 17:39:06.723719 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.723721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.724037 (+ 316us) server_negotiation.cc:407] Connection header received
0924 17:39:06.724260 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.724262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.724273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.724303 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.725437 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.726143 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.727205 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.727486 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.727523 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.729226 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.729227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.729228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.729236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.730660 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.731026 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.731027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.731028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.731119 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.731347 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.731348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.731348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.731487 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.731571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.731794 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.731870 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":36}
I20260924 17:39:06.732627 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.722647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.0.98.61:33309)
0924 17:39:06.722695 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.722697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.723703 (+ 1006us) server_negotiation.cc:407] Connection header received
0924 17:39:06.723903 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.723904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.723920 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.723952 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.724904 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.725698 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.727139 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.727447 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.727487 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.729228 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.729229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.729229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.729236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.731335 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.731758 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.731759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.731759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.731817 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.732019 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.732019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.732020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.732145 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.732216 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.732444 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.732534 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:39:06.732930 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:06.712136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.0.98.60:40275)
0924 17:39:06.712188 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:06.712191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:06.712729 (+ 538us) server_negotiation.cc:407] Connection header received
0924 17:39:06.722609 (+ 9880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:06.722611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:06.722629 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:06.722663 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:06.723623 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.724341 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.725495 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:06.725841 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:06.725889 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:06.727729 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:06.727730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:06.727730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:06.727737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:06.731595 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.731975 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.731976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.731976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.732024 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:06.732285 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:06.732286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:06.732286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:06.732401 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:06.732471 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:06.732688 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 17:39:06.732781 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":26}
I20260924 17:39:06.735428 7779 ts_tablet_manager.cc:1905] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:06.736989 7779 ts_tablet_manager.cc:1918] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:06.737088 7779 log.cc:1196] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/ec5349036eff40bcac6ae4f570c025e5
I20260924 17:39:06.737367 7779 ts_tablet_manager.cc:1939] T ec5349036eff40bcac6ae4f570c025e5 P 7ecb344160a643a39efa254fc861e47c: Deleting consensus metadata
W20260924 17:39:06.737756 7344 connection.cc:441] server connection from 127.0.0.1:51626 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51626 (request call id 1) could send its response
I20260924 17:39:07.707377 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:07.707043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:41883)
0924 17:39:07.707249 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:07.707333 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":56,"threads_started":1}
W20260924 17:39:07.707574 7730 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:07.708520 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:07.708203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:56657)
0924 17:39:07.708409 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:07.708474 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":52,"threads_started":1}
I20260924 17:39:07.708616 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:07.708320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:39317)
0924 17:39:07.708509 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:07.708572 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":49,"threads_started":1}
W20260924 17:39:07.708719 7594 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
W20260924 17:39:07.708829 7457 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.210997 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a57e962e8d03421597cf7b8e34bf4715)
I20260924 17:39:08.211247 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } }
I20260924 17:39:08.211539 7793 leader_election.cc:288] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945), 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60:40275)
I20260924 17:39:08.211928 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.211724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42866)
0924 17:39:08.211825 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.211879 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260924 17:39:08.212067 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a57e962e8d03421597cf7b8e34bf4715)
W20260924 17:39:08.212169 7069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.212201 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } }
I20260924 17:39:08.212651 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.212540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42884)
0924 17:39:08.212585 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.212616 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 17:39:08.212745 7069 leader_election.cc:334] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.212779 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945), be6c68f5902a4725af474b575d30b50c (127.0.98.61:33309)
I20260924 17:39:08.212682 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.212519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42882)
0924 17:39:08.212603 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.212648 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":39,"threads_started":1}
W20260924 17:39:08.213001 7261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.213431 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.213347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42900)
0924 17:39:08.213378 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.213402 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 17:39:08.213500 7261 leader_election.cc:334] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.215371 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.211895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40275 (local address 127.0.0.1:41380)
0924 17:39:08.211994 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.212005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:08.212057 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:08.212386 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:08.212387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:08.212390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:08.212447 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.212450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.213443 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.213444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.214575 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.214578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.215179 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.215180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.215239 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.215243 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:08.215284 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 17:39:08.215318 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260924 17:39:08.215411 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.212059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.0.98.60:40275)
0924 17:39:08.212137 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.212139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.212148 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:08.212213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.212214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.212230 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.212310 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:08.212534 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.213371 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.214662 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.215097 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.215171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.215182 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.215307 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.215348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"thread_start_us":34,"threads_started":1}
I20260924 17:39:08.215701 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be6c68f5902a4725af474b575d30b50c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" is_pre_election: true
I20260924 17:39:08.215812 7287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6c68f5902a4725af474b575d30b50c in term 1.
I20260924 17:39:08.215973 7069 leader_election.cc:302] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff6bf55e53e4b47afb763d68cd5eb8d, be6c68f5902a4725af474b575d30b50c; no voters: a57e962e8d03421597cf7b8e34bf4715
I20260924 17:39:08.216091 7793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 17:39:08.216140 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a57e962e8d03421597cf7b8e34bf4715)
I20260924 17:39:08.216168 7793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:39:08.216295 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.212710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33309 (local address 127.0.0.1:56314)
0924 17:39:08.212869 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.212878 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:08.212933 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:08.213179 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:08.213180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:08.213182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:08.213222 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.213224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.214035 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.214036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.215022 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.215024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.216035 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.216036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.216109 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.216113 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:08.216186 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:39:08.216230 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":45,"threads_started":1}
I20260924 17:39:08.216288 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.212831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.98.61:33309)
0924 17:39:08.213053 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.213055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.213063 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:08.213083 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.213083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.213092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.213157 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:08.213316 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.213953 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.215140 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.215887 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.216020 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.216029 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.216193 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.216231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260924 17:39:08.216979 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } }
I20260924 17:39:08.217123 7793 leader_election.cc:288] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 election: Requested vote from peers a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945), 3ff6bf55e53e4b47afb763d68cd5eb8d (127.0.98.60:40275)
I20260924 17:39:08.217294 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "be6c68f5902a4725af474b575d30b50c" is_pre_election: true
I20260924 17:39:08.217350 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be6c68f5902a4725af474b575d30b50c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d"
I20260924 17:39:08.217412 7095 raft_consensus.cc:2379] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ff6bf55e53e4b47afb763d68cd5eb8d in current term 2: Already voted for candidate be6c68f5902a4725af474b575d30b50c in this term.
I20260924 17:39:08.217432 7287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:39:08.217577 7263 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ff6bf55e53e4b47afb763d68cd5eb8d; no voters: a57e962e8d03421597cf7b8e34bf4715, be6c68f5902a4725af474b575d30b50c
I20260924 17:39:08.217304 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.217208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42902)
0924 17:39:08.217245 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.217280 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 17:39:08.218289 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.218196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42914)
0924 17:39:08.218241 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.218267 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:39:08.218295 7287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6c68f5902a4725af474b575d30b50c in term 2.
W20260924 17:39:08.218401 7069 leader_election.cc:334] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
I20260924 17:39:08.218425 7796 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:39:08.218480 7069 leader_election.cc:302] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff6bf55e53e4b47afb763d68cd5eb8d, be6c68f5902a4725af474b575d30b50c; no voters: a57e962e8d03421597cf7b8e34bf4715
I20260924 17:39:08.218590 7793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:39:08.218758 7793 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [term 2 LEADER]: Becoming Leader. State: Replica: be6c68f5902a4725af474b575d30b50c, State: Running, Role: LEADER
I20260924 17:39:08.218843 7793 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } }
I20260924 17:39:08.219225 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6c68f5902a4725af474b575d30b50c. Latest consensus state: current_term: 2 leader_uuid: "be6c68f5902a4725af474b575d30b50c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:08.219319 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:08.219486 7805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:08.219940 7805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:08.220029 7805 catalog_manager.cc:1259] Loaded cluster ID: 1327fdf0142145b3988bf9b3650c3d55
I20260924 17:39:08.220054 7805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:08.220196 7805 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:08.220252 7805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Loaded TSK: 0
I20260924 17:39:08.220345 7805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:08.277118 7084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:08.278767 7084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:08.279992 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.279897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42938)
0924 17:39:08.279932 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.279965 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 17:39:08.279978 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [term 2 FOLLOWER]: Refusing update from remote peer be6c68f5902a4725af474b575d30b50c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 17:39:08.280344 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.280242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.0.1:42940)
0924 17:39:08.280288 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.280314 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:39:08.280383 7793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 17:39:08.280537 7069 consensus_peers.cc:487] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c -> Peer a57e962e8d03421597cf7b8e34bf4715 (127.0.98.62:39945): Couldn't send request to peer a57e962e8d03421597cf7b8e34bf4715. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 17:39:08.281062 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6c68f5902a4725af474b575d30b50c. Latest consensus state: current_term: 2 leader_uuid: "be6c68f5902a4725af474b575d30b50c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:08.281147 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:39:08.281656 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "be6c68f5902a4725af474b575d30b50c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:08.281769 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:08.281862 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "be6c68f5902a4725af474b575d30b50c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a57e962e8d03421597cf7b8e34bf4715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } peers { permanent_uuid: "be6c68f5902a4725af474b575d30b50c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "3ff6bf55e53e4b47afb763d68cd5eb8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40275 } attrs { promote: false } } }
I20260924 17:39:08.281955 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff6bf55e53e4b47afb763d68cd5eb8d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:39:08.286954 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.0.98.1:42723)
0924 17:39:08.283902 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.283904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.283913 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:08.283938 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.283939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.283949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.284138 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:08.284175 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.284935 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286173 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286614 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286725 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.286833 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.286874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260924 17:39:08.286962 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42723 (local address 127.0.0.1:51642)
0924 17:39:08.283452 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.283463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:08.283504 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:08.284032 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:08.284032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:08.284033 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:08.284073 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.284075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.285009 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.285010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.286071 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.286072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286697 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.286764 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286768 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:08.286825 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:39:08.286887 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260924 17:39:08.287137 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.98.3:45857)
0924 17:39:08.283929 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.283931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.283939 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:08.283963 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.283964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.283974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.284031 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:08.284185 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.284676 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286113 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286710 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286776 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286789 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.286992 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.287068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260924 17:39:08.287139 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.98.2:36183)
0924 17:39:08.283839 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.283842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.283851 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:08.283880 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.283881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.283893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.284138 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:08.284175 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.284929 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.285943 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286522 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286935 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286948 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.287009 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.287073 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 17:39:08.287211 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36183 (local address 127.0.0.1:38414)
0924 17:39:08.283567 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.283577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:08.283618 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:08.284001 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:08.284002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:08.284005 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:08.284073 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.284075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.285038 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.285039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.285808 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.285811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286616 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.286740 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286744 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:08.286776 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 17:39:08.286836 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":4}
I20260924 17:39:08.287431 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.283496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45857 (local address 127.0.0.1:49934)
0924 17:39:08.283746 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.283755 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:08.283801 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:08.284078 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:08.284079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:08.284081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:08.284128 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.284130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.284773 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.284774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.286029 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:08.286032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.286821 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.286822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:08.286872 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.286876 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:08.286921 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:39:08.286960 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":25,"thread_start_us":68,"threads_started":1}
I20260924 17:39:08.287465 7527 tablet_service.cc:1467] Processing CreateTablet for tablet 6387912a90944edbb708589843cf1cae (DEFAULT_TABLE table=test-table [id=af7f2c3f8ba34d2983bc538ff11e3ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:08.287657 7527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6387912a90944edbb708589843cf1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:08.287721 7663 tablet_service.cc:1467] Processing CreateTablet for tablet 6387912a90944edbb708589843cf1cae (DEFAULT_TABLE table=test-table [id=af7f2c3f8ba34d2983bc538ff11e3ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:08.287851 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6387912a90944edbb708589843cf1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:08.287976 7390 tablet_service.cc:1467] Processing CreateTablet for tablet 6387912a90944edbb708589843cf1cae (DEFAULT_TABLE table=test-table [id=af7f2c3f8ba34d2983bc538ff11e3ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:08.288101 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6387912a90944edbb708589843cf1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:08.289261 7815 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Bootstrap starting.
I20260924 17:39:08.289426 7816 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Bootstrap starting.
I20260924 17:39:08.289451 7817 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: Bootstrap starting.
I20260924 17:39:08.289846 7817 tablet_bootstrap.cc:654] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:08.289881 7816 tablet_bootstrap.cc:654] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:08.290383 7815 tablet_bootstrap.cc:654] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:08.290436 7816 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: No bootstrap required, opened a new log
I20260924 17:39:08.290488 7816 ts_tablet_manager.cc:1397] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:08.290674 7817 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: No bootstrap required, opened a new log
I20260924 17:39:08.290634 7816 raft_consensus.cc:348] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.290721 7816 raft_consensus.cc:374] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:08.290728 7817 ts_tablet_manager.cc:1397] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:08.290755 7816 raft_consensus.cc:729] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d98c45d3ab430bb8e1962ce54a1a0c, State: Initialized, Role: FOLLOWER
I20260924 17:39:08.290807 7816 consensus_queue.cc:260] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.290838 7817 raft_consensus.cc:348] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.290913 7817 raft_consensus.cc:374] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:08.290933 7817 raft_consensus.cc:729] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecb344160a643a39efa254fc861e47c, State: Initialized, Role: FOLLOWER
I20260924 17:39:08.290977 7816 ts_tablet_manager.cc:1428] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:08.291206 7815 tablet_bootstrap.cc:492] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: No bootstrap required, opened a new log
I20260924 17:39:08.291306 7815 ts_tablet_manager.cc:1397] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:08.291482 7815 raft_consensus.cc:348] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.291558 7815 raft_consensus.cc:374] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:08.291581 7815 raft_consensus.cc:729] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e63bb4cad2449489560ab8e63f4e588, State: Initialized, Role: FOLLOWER
I20260924 17:39:08.291500 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.291263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:49689)
0924 17:39:08.291408 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.291452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260924 17:39:08.291652 7815 consensus_queue.cc:260] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.290995 7817 consensus_queue.cc:260] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.291823 7815 ts_tablet_manager.cc:1428] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:08.292021 7817 ts_tablet_manager.cc:1428] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:08.292598 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.292307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:52683)
0924 17:39:08.292520 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.292563 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":127,"threads_started":1}
I20260924 17:39:08.292711 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.292494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:40693)
0924 17:39:08.292624 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.292663 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260924 17:39:08.293161 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.293081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:47867)
0924 17:39:08.293110 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.293135 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 17:39:08.293567 7731 heartbeater.cc:503] Master 127.0.98.61:33309 was elected leader, sending a full tablet report...
I20260924 17:39:08.293613 7458 heartbeater.cc:503] Master 127.0.98.61:33309 was elected leader, sending a full tablet report...
I20260924 17:39:08.293902 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.293383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:34607)
0924 17:39:08.293719 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.293872 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
I20260924 17:39:08.295305 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.291938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:40963)
0924 17:39:08.295238 (+ 3300us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.295271 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3287,"thread_start_us":76,"threads_started":1}
W20260924 17:39:08.307835 7459 tablet.cc:2369] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:39:08.352104 7732 tablet.cc:2369] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:08.392607 7824 raft_consensus.cc:482] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:08.392720 7824 raft_consensus.cc:504] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.392904 7824 leader_election.cc:288] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183), 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:08.393157 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6387912a90944edbb708589843cf1cae" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" is_pre_election: true
I20260924 17:39:08.393167 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6387912a90944edbb708589843cf1cae" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e63bb4cad2449489560ab8e63f4e588" is_pre_election: true
I20260924 17:39:08.393254 7547 raft_consensus.cc:2454] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 0.
I20260924 17:39:08.393285 7683 raft_consensus.cc:2454] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 0.
I20260924 17:39:08.393422 7346 leader_election.cc:302] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d98c45d3ab430bb8e1962ce54a1a0c, 7ecb344160a643a39efa254fc861e47c; no voters:
I20260924 17:39:08.393541 7824 raft_consensus.cc:2781] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:08.393601 7824 raft_consensus.cc:482] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:08.393617 7824 raft_consensus.cc:3037] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:08.394388 7824 raft_consensus.cc:504] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.394529 7824 leader_election.cc:288] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 election: Requested vote from peers 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183), 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857)
I20260924 17:39:08.394677 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6387912a90944edbb708589843cf1cae" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c"
I20260924 17:39:08.394769 7547 raft_consensus.cc:3037] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:08.394739 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6387912a90944edbb708589843cf1cae" candidate_uuid: "7ecb344160a643a39efa254fc861e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e63bb4cad2449489560ab8e63f4e588"
I20260924 17:39:08.394833 7683 raft_consensus.cc:3037] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:08.395522 7547 raft_consensus.cc:2454] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 1.
I20260924 17:39:08.395529 7683 raft_consensus.cc:2454] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb344160a643a39efa254fc861e47c in term 1.
I20260924 17:39:08.395749 7343 leader_election.cc:302] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e63bb4cad2449489560ab8e63f4e588, 7ecb344160a643a39efa254fc861e47c; no voters:
I20260924 17:39:08.395851 7824 raft_consensus.cc:2781] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:08.396018 7824 raft_consensus.cc:686] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [term 1 LEADER]: Becoming Leader. State: Replica: 7ecb344160a643a39efa254fc861e47c, State: Running, Role: LEADER
I20260924 17:39:08.396090 7824 consensus_queue.cc:237] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } }
I20260924 17:39:08.396382 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.396266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:37633)
0924 17:39:08.396320 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.396352 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:39:08.396693 7085 catalog_manager.cc:5495] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ecb344160a643a39efa254fc861e47c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7ecb344160a643a39efa254fc861e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecb344160a643a39efa254fc861e47c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 } health_report { overall_health: UNKNOWN } } }
I20260924 17:39:08.407559 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.405375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.98.1:42723)
0924 17:39:08.405423 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:08.405427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:08.405440 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:39:08.405465 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:08.405466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:08.405477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:08.405525 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:08.405631 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.406089 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.406814 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:08.407013 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:08.407122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:08.407260 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:08.407288 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:08.407325 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:08.407437 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 17:39:08.407457 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:39:08.408509 7683 raft_consensus.cc:1261] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Refusing update from remote peer 7ecb344160a643a39efa254fc861e47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:08.408520 7547 raft_consensus.cc:1261] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Refusing update from remote peer 7ecb344160a643a39efa254fc861e47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:08.408753 7824 consensus_queue.cc:1035] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28d98c45d3ab430bb8e1962ce54a1a0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:08.408864 7828 consensus_queue.cc:1035] T 6387912a90944edbb708589843cf1cae P 7ecb344160a643a39efa254fc861e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e63bb4cad2449489560ab8e63f4e588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:08.409235 7595 heartbeater.cc:503] Master 127.0.98.61:33309 was elected leader, sending a full tablet report...
I20260924 17:39:08.409709 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.409539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:40703)
0924 17:39:08.409610 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.409652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:39:08.409852 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.409548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:47645)
0924 17:39:08.409784 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.409815 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260924 17:39:08.410926 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.410711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:50205)
0924 17:39:08.410855 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.410889 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 17:39:08.411454 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.411365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:42947)
0924 17:39:08.411396 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.411423 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 17:39:08.411842 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.411251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:48467)
0924 17:39:08.411786 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.411815 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518}
I20260924 17:39:08.412016 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.411721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:59077)
0924 17:39:08.411956 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.411985 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260924 17:39:08.412392 7083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:08.412483 7083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:08.413800 7083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be6c68f5902a4725af474b575d30b50c: Sending DeleteTablet for 3 replicas of tablet 6387912a90944edbb708589843cf1cae
I20260924 17:39:08.414086 7527 tablet_service.cc:1514] Processing DeleteTablet for tablet 6387912a90944edbb708589843cf1cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:08 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38414
I20260924 17:39:08.414126 7390 tablet_service.cc:1514] Processing DeleteTablet for tablet 6387912a90944edbb708589843cf1cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:08 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51642
I20260924 17:39:08.414259 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet 6387912a90944edbb708589843cf1cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:08 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49934
I20260924 17:39:08.414428 7842 tablet_replica.cc:330] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: stopping tablet replica
I20260924 17:39:08.414435 7840 tablet_replica.cc:330] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: stopping tablet replica
I20260924 17:39:08.414505 7840 raft_consensus.cc:2229] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:08.414507 7842 raft_consensus.cc:2229] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:08.414515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7236
I20260924 17:39:08.414561 7842 raft_consensus.cc:2258] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:08.414561 7840 raft_consensus.cc:2258] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:08.415242 7840 ts_tablet_manager.cc:1905] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:08.415266 7842 ts_tablet_manager.cc:1905] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:08.416725 7842 ts_tablet_manager.cc:1918] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:08.416801 7842 log.cc:1196] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/6387912a90944edbb708589843cf1cae
I20260924 17:39:08.417026 7842 ts_tablet_manager.cc:1939] T 6387912a90944edbb708589843cf1cae P 3e63bb4cad2449489560ab8e63f4e588: Deleting consensus metadata
I20260924 17:39:08.417132 7840 ts_tablet_manager.cc:1918] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:08.417194 7840 log.cc:1196] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/6387912a90944edbb708589843cf1cae
I20260924 17:39:08.417332 7069 catalog_manager.cc:4841] TS 3e63bb4cad2449489560ab8e63f4e588 (127.0.98.3:45857): tablet 6387912a90944edbb708589843cf1cae (table test-table [id=af7f2c3f8ba34d2983bc538ff11e3ef3]) successfully deleted
I20260924 17:39:08.417436 7840 ts_tablet_manager.cc:1939] T 6387912a90944edbb708589843cf1cae P 28d98c45d3ab430bb8e1962ce54a1a0c: Deleting consensus metadata
I20260924 17:39:08.418222 7072 catalog_manager.cc:4841] TS 28d98c45d3ab430bb8e1962ce54a1a0c (127.0.98.2:36183): tablet 6387912a90944edbb708589843cf1cae (table test-table [id=af7f2c3f8ba34d2983bc538ff11e3ef3]) successfully deleted
I20260924 17:39:08.422875 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:08.422724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42723 (local address 127.0.0.1:51668)
0924 17:39:08.422799 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:08.422830 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 17:39:08.422878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7464
W20260924 17:39:08.423506 7069 catalog_manager.cc:4584] TS 7ecb344160a643a39efa254fc861e47c (127.0.98.1:42723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6387912a90944edbb708589843cf1cae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42723: connect: Connection refused (error 111)
I20260924 17:39:08.428750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7600
I20260924 17:39:08.433351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 6950
I20260924 17:39:08.438082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7139
2026-09-24T17:39:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4715 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:08 dist-test-slave-d0pf krb5kdc[7845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:08 dist-test-slave-d0pf krb5kdc[7845](info): set up 2 sockets
Sep 24 17:39:08 dist-test-slave-d0pf krb5kdc[7845](info): commencing operation
krb5kdc: starting...
W20260924 17:39:09.483960 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:09 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 17:39:09.601549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:09.682066 7858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:09.682274 7858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:09.682296 7858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:09.683535 7858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:09.683574 7858 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:39:09.683588 7858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:09.683631 7858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:09.683658 7858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:09.683674 7858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:09.685127 7858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39407
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:09.685384 7858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:09.685617 7858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:09.690517 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:09.690492 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:09.690498 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:09.690786 7858 server_base.cc:1034] running on GCE node
I20260924 17:39:09.691154 7858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:09.691460 7858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:09.692600 7858 hybrid_clock.cc:648] HybridClock initialized: now 1790271549692589 us; error 29 us; skew 500 ppm
Sep 24 17:39:09 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:09.695173 7858 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 17:39:09.695672 7858 webserver.cc:460] Webserver started at http://127.0.98.62:36429/ using document root <none> and password file <none>
I20260924 17:39:09.695940 7858 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:09.696020 7858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:09.696156 7858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:09.697228 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "79887010157c4c36ac79fd72d702c31a"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "f8d58656c72111f2cb9ddd42524d2226"
server_key_iv: "c25bb8efc4c77a51e20ce86b13a96b6d"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.697611 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "79887010157c4c36ac79fd72d702c31a"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "f8d58656c72111f2cb9ddd42524d2226"
server_key_iv: "c25bb8efc4c77a51e20ce86b13a96b6d"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.698927 7858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:09.699710 7873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:09.699904 7858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:39:09.699993 7858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "79887010157c4c36ac79fd72d702c31a"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "f8d58656c72111f2cb9ddd42524d2226"
server_key_iv: "c25bb8efc4c77a51e20ce86b13a96b6d"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.700064 7858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:09.713840 7858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:09.716308 7858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:09.716461 7858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:09.721160 7858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39407
I20260924 17:39:09.721202 7935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39407 every 8 connection(s)
I20260924 17:39:09.721531 7858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:09.722147 7936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:09.724450 7936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Bootstrap starting.
I20260924 17:39:09.725029 7936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:09.725322 7936 log.cc:824] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:09.725983 7936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: No bootstrap required, opened a new log
I20260924 17:39:09.726444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7858
I20260924 17:39:09.726524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:09.726723 392 external_mini_cluster.cc:1428] Setting key d2ffac7ced0b3bd8e1b7f7687867080c
I20260924 17:39:09.728768 7936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } }
I20260924 17:39:09.728978 7936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:09.729028 7936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79887010157c4c36ac79fd72d702c31a, State: Initialized, Role: FOLLOWER
I20260924 17:39:09.729188 7936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } }
I20260924 17:39:09.729274 7936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:09.729310 7936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:09.729362 7936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:09.730027 7936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } }
I20260924 17:39:09.730152 7936 leader_election.cc:302] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79887010157c4c36ac79fd72d702c31a; no voters:
I20260924 17:39:09.730316 7936 leader_election.cc:288] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:09.730348 7941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:09.730468 7941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [term 1 LEADER]: Becoming Leader. State: Replica: 79887010157c4c36ac79fd72d702c31a, State: Running, Role: LEADER
I20260924 17:39:09.730553 7941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } }
I20260924 17:39:09.730580 7936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:09.730883 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79887010157c4c36ac79fd72d702c31a. Latest consensus state: current_term: 1 leader_uuid: "79887010157c4c36ac79fd72d702c31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } } }
I20260924 17:39:09.730952 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:09.730891 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79887010157c4c36ac79fd72d702c31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79887010157c4c36ac79fd72d702c31a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39407 } } }
I20260924 17:39:09.731178 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:09.731132 7945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:09.732277 7945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:09.733911 7945 catalog_manager.cc:1347] Generated new cluster ID: 35f573eecbf745acaf8abe6f2df73fa8
I20260924 17:39:09.733963 7945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:39:09 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:09.739198 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:09.727283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.0.98.62:39407)
0924 17:39:09.727502 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:09.727511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:09.727546 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:39:09.728692 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:09.728703 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:09.728774 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:09.728952 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:09.729827 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.731171 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.732289 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.732779 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.732918 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:09.735489 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:09.735492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:09.735493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:09.735505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:09.737631 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.738024 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.738027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.738027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.738117 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.738337 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.738338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.738339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.738582 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:09.738765 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:09.738839 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 17:39:09.738974 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":110,"threads_started":1}
I20260924 17:39:09.778203 7945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:09.778820 7945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:09.783231 7945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Generated new TSK 0
I20260924 17:39:09.783466 7945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 17:39:09.822389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39407
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:09.941612 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:09.941848 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:09.941876 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:09.943135 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:09.943176 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:09.943198 7962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:39:09.943217 7962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:09.944790 7962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:09.944995 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:09.945199 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:09.950815 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:09.950819 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:09.950871 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:09.951225 7962 server_base.cc:1034] running on GCE node
I20260924 17:39:09.951471 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:09.951769 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:09.952942 7962 hybrid_clock.cc:648] HybridClock initialized: now 1790271549952922 us; error 33 us; skew 500 ppm
Sep 24 17:39:09 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:09.955595 7962 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 17:39:09.955989 7962 webserver.cc:460] Webserver started at http://127.0.98.1:40303/ using document root <none> and password file <none>
I20260924 17:39:09.956183 7962 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:09.956223 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:09.956307 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:09.957307 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e5681b9e18e415ca9778a4ecfd98601"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "c0d9359f1b3e59329dbb885795a7611f"
server_key_iv: "7028ba5cf81735dbe2eeb381c81299c8"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.957604 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e5681b9e18e415ca9778a4ecfd98601"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "c0d9359f1b3e59329dbb885795a7611f"
server_key_iv: "7028ba5cf81735dbe2eeb381c81299c8"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.958969 7962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:39:09.959745 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:09.959914 7962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:09.960019 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "0e5681b9e18e415ca9778a4ecfd98601"
format_stamp: "Formatted at 2026-09-24 17:39:09 on dist-test-slave-d0pf"
server_key: "c0d9359f1b3e59329dbb885795a7611f"
server_key_iv: "7028ba5cf81735dbe2eeb381c81299c8"
server_key_version: "encryptionkey@0"
I20260924 17:39:09.960094 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:09.977770 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:09.980172 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:09.980341 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:09.980866 7962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:09.980903 7962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:09.980929 7962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:09.980947 7962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:39:09 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:09.987923 7962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36163
I20260924 17:39:09.988058 8092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36163 every 8 connection(s)
I20260924 17:39:09.988329 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:39:09.990990 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:09.981444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60061 (local address 127.0.98.62:39407)
0924 17:39:09.981489 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:09.981491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:09.981735 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:39:09.982318 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:09.982320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:09.982336 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:09.982402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:09.983352 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.983855 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.985496 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.985862 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.985934 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:09.988479 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:09.988481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:09.988481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:09.988491 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:09.989824 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.990191 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.990191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.990192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.990240 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.990446 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.990447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.990447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.990547 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:09.990596 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:09.990843 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 17:39:09.990910 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:09.991209 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:09.981447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.1:60061)
0924 17:39:09.981615 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:09.981649 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:09.982167 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:09.982538 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:09.982543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:09.982931 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:09.983264 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:09.983271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.983950 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.983951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:09.985390 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:09.985394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.985967 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.985968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:09.986097 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:09.986625 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:09.986633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:09.988370 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:09.989891 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:09.989896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:09.989898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:09.990125 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:09.990297 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:09.990297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:09.990298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:09.990392 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:09.990602 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:09.990605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:09.990735 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 17:39:09.990817 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:39:09.990941 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260924 17:39:09.991705 8093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39407
I20260924 17:39:09.991792 8093 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:09.991979 8093 heartbeater.cc:511] Master 127.0.98.62:39407 requested a full tablet report, sending...
I20260924 17:39:09.992563 7889 ts_manager.cc:194] Registered new tserver with Master: 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163)
I20260924 17:39:09.993530 7889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60061
I20260924 17:39:09.998024 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:09.992892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48837 (local address 127.0.98.62:39407)
0924 17:39:09.992938 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:09.992940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:09.993031 (+ 91us) server_negotiation.cc:407] Connection header received
0924 17:39:09.993099 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:09.993099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:09.993107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:09.993149 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:09.993503 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.994104 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.994819 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.995004 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.995137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:09.996022 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:09.996023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:09.996023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:09.996029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:09.997168 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.997463 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.997464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.997464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.997511 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:09.997664 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:09.997664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:09.997665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:09.997755 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:09.997802 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:09.997902 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 17:39:09.997966 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:39:09.998034 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:09.992801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.1:48837)
0924 17:39:09.992973 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:09.992984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:09.993046 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:09.993162 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:09.993162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:09.993350 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:09.993407 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:09.993408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.994188 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.994189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:09.994747 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:09.994750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:09.995133 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:09.995134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:09.995187 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:09.995649 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:09.995654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:09.995948 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:09.997241 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:09.997242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:09.997242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:09.997401 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:09.997564 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:09.997565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:09.997565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:09.997614 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:09.997804 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:09.997805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:09.997838 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 17:39:09.997886 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 17:39:09.997960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260924 17:39:09.998533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 7962
I20260924 17:39:09.998636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:09.998842 392 external_mini_cluster.cc:1428] Setting key eaf31fb531147318b791a27dbf8d4b35
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 17:39:10.023736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39407
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:10.105537 8102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:10.105753 8102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:10.105777 8102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:10.106945 8102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:10.106988 8102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:10.107009 8102 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:39:10.107028 8102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:10.108546 8102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:10.108783 8102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:10.109016 8102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:10.114147 8108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:10.114169 8107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:10.114243 8102 server_base.cc:1034] running on GCE node
W20260924 17:39:10.114147 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:10.114773 8102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:10.115096 8102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:10.116253 8102 hybrid_clock.cc:648] HybridClock initialized: now 1790271550116233 us; error 38 us; skew 500 ppm
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:10.124984 8102 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 17:39:10.125517 8102 webserver.cc:460] Webserver started at http://127.0.98.2:41891/ using document root <none> and password file <none>
I20260924 17:39:10.125763 8102 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:10.125844 8102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:10.125998 8102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:10.127211 8102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "022951de34a141a9a170f647e8941675"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "7b867386fbc134730b5283e54a9b50a1"
server_key_iv: "fe5212b0e3253ab0b0b5cdc9da10df27"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.127601 8102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "022951de34a141a9a170f647e8941675"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "7b867386fbc134730b5283e54a9b50a1"
server_key_iv: "fe5212b0e3253ab0b0b5cdc9da10df27"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.129000 8102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:10.129807 8117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.130038 8102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.130123 8102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "022951de34a141a9a170f647e8941675"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "7b867386fbc134730b5283e54a9b50a1"
server_key_iv: "fe5212b0e3253ab0b0b5cdc9da10df27"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.130179 8102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:10.153928 8102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:10.156064 8102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:10.156181 8102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:10.156713 8102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:10.156783 8102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.156822 8102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:10.156872 8102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.164389 8102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38623
I20260924 17:39:10.164459 8232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38623 every 8 connection(s)
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:10.164799 8102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:10.168426 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.157426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40909 (local address 127.0.98.62:39407)
0924 17:39:10.157498 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.157500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.157704 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:39:10.158267 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.158268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.158285 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.158374 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.159352 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.160245 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.161906 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.162275 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.162379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.165473 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.165475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.165475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.165483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.166850 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.167357 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.167358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.167359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.167418 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.167757 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.167758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.167758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.167870 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.167920 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.168253 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.168331 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:39:10.168555 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.157384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.2:40909)
0924 17:39:10.157565 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.157608 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.158153 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.158479 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.158484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.158869 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:10.159245 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.159251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.160353 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.160354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.161793 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.161796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.162391 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.162392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.162529 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.163243 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:10.163257 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:10.165357 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:10.166951 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.166957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.166959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.167266 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.167550 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.167551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.167552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.167682 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.167929 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:10.167934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:10.168073 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.168170 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.168311 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260924 17:39:10.168735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8102
I20260924 17:39:10.168849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:10.169082 392 external_mini_cluster.cc:1428] Setting key 51ac59acd1eb1e592178a9cf60b17a8b
I20260924 17:39:10.170280 8233 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39407
I20260924 17:39:10.170374 8233 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:10.170574 8233 heartbeater.cc:511] Master 127.0.98.62:39407 requested a full tablet report, sending...
I20260924 17:39:10.171000 7890 ts_manager.cc:194] Registered new tserver with Master: 022951de34a141a9a170f647e8941675 (127.0.98.2:38623)
I20260924 17:39:10.171597 7890 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40909
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:39:10.177205 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.171216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41777 (local address 127.0.98.62:39407)
0924 17:39:10.171267 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.171269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.171277 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:10.171296 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.171297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.171307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.171385 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.171761 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.172452 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.173238 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.173446 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.173538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.174614 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.174615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.174616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.174623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.176003 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.176366 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.176368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.176368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.176433 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.176638 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.176639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.176639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.176797 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.176877 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.177027 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.177109 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:39:10.177219 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.170422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.2:41777)
0924 17:39:10.170576 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.170589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.170645 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.171391 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.171392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.171586 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:10.171637 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.171645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.172570 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.172571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.173160 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.173162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.173540 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.173540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.173607 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.174167 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:10.174172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:10.174485 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:10.176092 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.176093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.176094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.176278 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.176504 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.176504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.176505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.176565 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.176868 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:10.176869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:10.176926 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.176989 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.177107 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 17:39:10.194133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39407
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:10.281334 8242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:10.281613 8242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:10.281661 8242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:10.283022 8242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:10.283136 8242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:10.283176 8242 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:39:10.283224 8242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:10.284946 8242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:10.285269 8242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:10.285537 8242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:10.291178 8248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:10.291178 8250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:10.291199 8247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:10.291690 8242 server_base.cc:1034] running on GCE node
I20260924 17:39:10.291975 8242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:10.292317 8242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:10.293497 8242 hybrid_clock.cc:648] HybridClock initialized: now 1790271550293491 us; error 40 us; skew 500 ppm
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:10.296293 8242 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 17:39:10.296857 8242 webserver.cc:460] Webserver started at http://127.0.98.3:38777/ using document root <none> and password file <none>
I20260924 17:39:10.297125 8242 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:10.297204 8242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:10.297366 8242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:10.298544 8242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "840fc1de614d444b82ea9c032d4ab647"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "4a6e305ae8feb6f1ea5d27bf1053f979"
server_key_iv: "a6813228b5fa812b51ef1f7daaff9e98"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.298960 8242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "840fc1de614d444b82ea9c032d4ab647"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "4a6e305ae8feb6f1ea5d27bf1053f979"
server_key_iv: "a6813228b5fa812b51ef1f7daaff9e98"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.300426 8242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:10.301146 8257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.301318 8242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.301394 8242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "840fc1de614d444b82ea9c032d4ab647"
format_stamp: "Formatted at 2026-09-24 17:39:10 on dist-test-slave-d0pf"
server_key: "4a6e305ae8feb6f1ea5d27bf1053f979"
server_key_iv: "a6813228b5fa812b51ef1f7daaff9e98"
server_key_version: "encryptionkey@0"
I20260924 17:39:10.301477 8242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:10.309628 8242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:10.312314 8242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:10.312456 8242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:10.313220 8242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:10.313278 8242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.313314 8242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:10.313338 8242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:10.321424 8242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41207
I20260924 17:39:10.321568 8372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41207 every 8 connection(s)
I20260924 17:39:10.321911 8242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:10.324337 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.313722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41395 (local address 127.0.98.62:39407)
0924 17:39:10.313801 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.313805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.314109 (+ 304us) server_negotiation.cc:407] Connection header received
0924 17:39:10.314869 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.314870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.314882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.314947 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.315950 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.316647 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.317801 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.318009 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.318098 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.320966 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.320968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.320968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.320978 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.322765 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.323279 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.323280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.323280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.323338 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.323677 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.323678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.323679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.323812 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.323871 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.324195 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.324271 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:39:10.324501 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.313746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.3:41395)
0924 17:39:10.313940 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.313991 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.314750 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.315066 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.315071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.315493 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:10.315832 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.315840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.316788 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.316789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.317701 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.317704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.318107 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.318108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.318251 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.318845 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:10.318854 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:10.320849 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:10.322853 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.322857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.322860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.323196 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.323450 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.323451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.323451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.323568 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.323897 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:10.323901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:10.324029 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.324122 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.324258 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260924 17:39:10.325027 8373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39407
I20260924 17:39:10.325114 8373 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:10.325307 8373 heartbeater.cc:511] Master 127.0.98.62:39407 requested a full tablet report, sending...
I20260924 17:39:10.325716 7889 ts_manager.cc:194] Registered new tserver with Master: 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207)
I20260924 17:39:10.326236 7889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41395
I20260924 17:39:10.329303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8242
I20260924 17:39:10.329376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:10.329599 392 external_mini_cluster.cc:1428] Setting key 60441a70c2d49cdbc0770d953a79d353
I20260924 17:39:10.330475 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:39:10.332098 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.326129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49901 (local address 127.0.98.62:39407)
0924 17:39:10.326169 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.326171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.326246 (+ 75us) server_negotiation.cc:407] Connection header received
0924 17:39:10.326320 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.326321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.326332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.326389 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.326786 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.327344 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.328076 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.328264 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.328332 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.329206 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.329208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.329208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.329216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.330893 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.331311 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.331312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.331312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.331368 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.331577 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.331578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.331579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.331702 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.331778 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.331935 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.332008 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:39:10.332146 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.326046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.98.3:49901)
0924 17:39:10.326177 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.326187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.326248 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.326395 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.326396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.326626 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:10.326702 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.326704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.327443 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.327444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.327979 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.327981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.328343 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.328343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.328435 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.328845 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:10.328849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:10.329118 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:10.331004 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.331006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.331006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.331237 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.331451 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.331452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.331452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.331515 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.331769 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:10.331770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:10.331814 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.331908 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.332010 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:39:10.352175 7940 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:10.346821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.98.62:39407)
0924 17:39:10.346906 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.346908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.346918 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:10.346951 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.346952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.346961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.347015 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.347632 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.348118 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.348854 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.349032 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.349083 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.352078 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58446: BlockingRecv error: recv got EOF from 127.0.0.1:58446 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:10.359666 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.352943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.0.98.62:39407)
0924 17:39:10.353001 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.353003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.353046 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:39:10.353109 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.353110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.353117 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.353168 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.353622 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.354098 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.354767 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.354949 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.355005 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.357296 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.357297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.357297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.357304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.358542 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.358846 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.358848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.358848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.358900 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.359201 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.359204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.359204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.359318 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.359395 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.359537 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.359597 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:39:10.361294 7889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:10.362900 7889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:10.369758 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36163 (local address 127.0.0.1:48864)
0924 17:39:10.365632 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.365646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.365717 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.366021 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.366025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.366027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.366112 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.366116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367465 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.367466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.368495 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.368498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369438 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.369517 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.369531 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.369610 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.369665 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
I20260924 17:39:10.369800 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.0.98.1:36163)
0924 17:39:10.365821 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.365826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.365837 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:10.365878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.365880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.365896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.365991 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.366206 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367379 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.368593 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369363 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.369520 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.369637 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.369687 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 17:39:10.369958 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.98.3:41207)
0924 17:39:10.365820 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.365826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.365865 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:39:10.365938 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.365940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.365955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.366025 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.366207 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367355 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.368808 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369406 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369474 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.369557 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.369793 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.369835 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260924 17:39:10.370164 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.0.1:59306)
0924 17:39:10.365820 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.365827 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.365878 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.366033 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.366033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.366035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.366112 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.366115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367436 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.367436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.368714 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.368716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369590 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.369657 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.369660 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.369703 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.369738 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":52,"thread_start_us":105,"threads_started":1}
I20260924 17:39:10.370522 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.0.1:35658)
0924 17:39:10.365570 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.365646 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.365717 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.366347 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.366348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.366349 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.366404 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.366406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367331 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.367332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.368699 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.368702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369796 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.370271 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.370278 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.370387 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.370442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":43,"thread_start_us":54,"threads_started":1}
I20260924 17:39:10.370565 8027 tablet_service.cc:1467] Processing CreateTablet for tablet b0b42889682848d8a61de8f4d6e24f2d (DEFAULT_TABLE table=test-table [id=81de98e014154e6f9d5b8c2a877451da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.370815 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.365481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.98.2:38623)
0924 17:39:10.366130 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.366134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.366144 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:10.366178 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.366181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.366195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.366277 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.366473 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.367236 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.368798 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.369621 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.369688 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.369761 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.370662 (+ 901us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.370721 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":82,"threads_started":1}
I20260924 17:39:10.370803 8307 tablet_service.cc:1467] Processing CreateTablet for tablet b0b42889682848d8a61de8f4d6e24f2d (DEFAULT_TABLE table=test-table [id=81de98e014154e6f9d5b8c2a877451da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.370962 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b42889682848d8a61de8f4d6e24f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.371132 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b42889682848d8a61de8f4d6e24f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.371435 8167 tablet_service.cc:1467] Processing CreateTablet for tablet b0b42889682848d8a61de8f4d6e24f2d (DEFAULT_TABLE table=test-table [id=81de98e014154e6f9d5b8c2a877451da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.371752 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b42889682848d8a61de8f4d6e24f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.374768 8401 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Bootstrap starting.
I20260924 17:39:10.375200 8402 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Bootstrap starting.
I20260924 17:39:10.375263 8403 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Bootstrap starting.
I20260924 17:39:10.375674 8401 tablet_bootstrap.cc:654] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.375775 8402 tablet_bootstrap.cc:654] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.375834 8403 tablet_bootstrap.cc:654] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.376034 8401 log.cc:824] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:10.376107 8402 log.cc:824] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:10.376190 8403 log.cc:824] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:10.376832 8401 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: No bootstrap required, opened a new log
I20260924 17:39:10.376837 8402 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: No bootstrap required, opened a new log
I20260924 17:39:10.376902 8401 ts_tablet_manager.cc:1397] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:39:10.376904 8402 ts_tablet_manager.cc:1397] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:10.378288 8402 raft_consensus.cc:348] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.378288 8401 raft_consensus.cc:348] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.378439 8402 raft_consensus.cc:374] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.378463 8402 raft_consensus.cc:729] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022951de34a141a9a170f647e8941675, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.378463 8401 raft_consensus.cc:374] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.378506 8401 raft_consensus.cc:729] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.378588 8402 consensus_queue.cc:260] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.378618 8401 consensus_queue.cc:260] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.378867 8402 ts_tablet_manager.cc:1428] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:39:10.378885 8401 ts_tablet_manager.cc:1428] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:39:10.378917 8373 heartbeater.cc:503] Master 127.0.98.62:39407 was elected leader, sending a full tablet report...
I20260924 17:39:10.378973 8233 heartbeater.cc:503] Master 127.0.98.62:39407 was elected leader, sending a full tablet report...
I20260924 17:39:10.379350 8403 tablet_bootstrap.cc:492] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: No bootstrap required, opened a new log
I20260924 17:39:10.379429 8403 ts_tablet_manager.cc:1397] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 17:39:10.381129 8403 raft_consensus.cc:348] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.381327 8403 raft_consensus.cc:374] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.381371 8403 raft_consensus.cc:729] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.381493 8403 consensus_queue.cc:260] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.381798 8403 ts_tablet_manager.cc:1428] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:39:10.382017 8093 heartbeater.cc:503] Master 127.0.98.62:39407 was elected leader, sending a full tablet report...
W20260924 17:39:10.415457 8234 tablet.cc:2369] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:39:10.489056 8094 tablet.cc:2369] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:10.549600 8407 raft_consensus.cc:482] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:10.549722 8407 raft_consensus.cc:504] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.550136 8407 leader_election.cc:288] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207), 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163)
I20260924 17:39:10.553960 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.550463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52363 (local address 127.0.98.1:36163)
0924 17:39:10.550507 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.550509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.550559 (+ 50us) server_negotiation.cc:407] Connection header received
0924 17:39:10.550584 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.550585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.550597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.550670 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.550896 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.551652 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553197 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.553573 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553658 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.553668 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.553835 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.553877 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:39:10.553969 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.550462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53479 (local address 127.0.98.3:41207)
0924 17:39:10.550507 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.550509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.550646 (+ 137us) server_negotiation.cc:407] Connection header received
0924 17:39:10.550671 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.550672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.550683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.550739 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.550881 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.551685 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553218 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.553574 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553681 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.553691 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.553859 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.553897 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:39:10.553992 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.550377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36163 (local address 127.0.98.2:52363)
0924 17:39:10.550440 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.550453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.550503 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.550703 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.550704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.550706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.550800 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.550802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.551780 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.551781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.553078 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.553082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.553660 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.553744 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.553747 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.553825 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.553877 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":14}
I20260924 17:39:10.554005 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.550400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.98.2:53479)
0924 17:39:10.550552 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.550560 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.550612 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.550749 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.550749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.550750 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.550813 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.550815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.551780 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.551781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.553078 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.553083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.553686 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.553687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.553773 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.553777 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.553851 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.553897 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":61,"thread_start_us":92,"threads_started":1}
I20260924 17:39:10.554391 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b42889682848d8a61de8f4d6e24f2d" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601" is_pre_election: true
I20260924 17:39:10.554596 8047 raft_consensus.cc:2454] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 0.
I20260924 17:39:10.554590 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b42889682848d8a61de8f4d6e24f2d" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647" is_pre_election: true
I20260924 17:39:10.554756 8327 raft_consensus.cc:2454] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 0.
I20260924 17:39:10.554872 8119 leader_election.cc:302] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022951de34a141a9a170f647e8941675, 0e5681b9e18e415ca9778a4ecfd98601; no voters:
I20260924 17:39:10.554996 8407 raft_consensus.cc:2781] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:10.555120 8407 raft_consensus.cc:482] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:10.555148 8407 raft_consensus.cc:3037] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.556104 8407 raft_consensus.cc:504] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.556250 8407 leader_election.cc:288] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 election: Requested vote from peers 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207), 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163)
I20260924 17:39:10.556391 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b42889682848d8a61de8f4d6e24f2d" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647"
I20260924 17:39:10.556447 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b42889682848d8a61de8f4d6e24f2d" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601"
I20260924 17:39:10.556494 8327 raft_consensus.cc:3037] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.556556 8047 raft_consensus.cc:3037] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.557221 8327 raft_consensus.cc:2454] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 1.
I20260924 17:39:10.557227 8047 raft_consensus.cc:2454] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 1.
I20260924 17:39:10.557415 8121 leader_election.cc:302] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022951de34a141a9a170f647e8941675, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:10.557521 8407 raft_consensus.cc:2781] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:10.557686 8407 raft_consensus.cc:686] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 1 LEADER]: Becoming Leader. State: Replica: 022951de34a141a9a170f647e8941675, State: Running, Role: LEADER
I20260924 17:39:10.557796 8407 consensus_queue.cc:237] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } }
I20260924 17:39:10.558565 7889 catalog_manager.cc:5495] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022951de34a141a9a170f647e8941675 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } health_report { overall_health: UNKNOWN } } }
W20260924 17:39:10.572583 8374 tablet.cc:2369] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:10.596285 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.593813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.0.98.2:38623)
0924 17:39:10.593855 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.593857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.593899 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:39:10.593946 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.593947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.593966 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.594042 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:10.594207 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.594675 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.595503 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.595701 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.595778 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.595952 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.596056 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.596149 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.596174 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.596193 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:10.598145 8327 raft_consensus.cc:1261] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Refusing update from remote peer 022951de34a141a9a170f647e8941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.598145 8047 raft_consensus.cc:1261] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Refusing update from remote peer 022951de34a141a9a170f647e8941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.598528 8407 consensus_queue.cc:1035] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.598701 8411 consensus_queue.cc:1035] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.600999 8414 mvcc.cc:205] Tried to move back new op lower bound from 7332952271246835712 to 7332952271085137920. Current Snapshot: MvccSnapshot[applied={T|T < 7332952271246835712}]
I20260924 17:39:10.601426 8416 mvcc.cc:205] Tried to move back new op lower bound from 7332952271246835712 to 7332952271085137920. Current Snapshot: MvccSnapshot[applied={T|T < 7332952271246835712}]
I20260924 17:39:10.604039 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:10.604147 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:10.604959 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Sending DeleteTablet for 3 replicas of tablet b0b42889682848d8a61de8f4d6e24f2d
I20260924 17:39:10.605275 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b42889682848d8a61de8f4d6e24f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35658
I20260924 17:39:10.605302 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b42889682848d8a61de8f4d6e24f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260924 17:39:10.605346 8027 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b42889682848d8a61de8f4d6e24f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 17:39:10.605580 8424 tablet_replica.cc:330] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: stopping tablet replica
I20260924 17:39:10.605717 8424 raft_consensus.cc:2229] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:10.605719 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:10.605896 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:10.605989 8424 raft_consensus.cc:2258] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.606582 8424 ts_tablet_manager.cc:1905] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.607223 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 (DEFAULT_TABLE table=test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.607221 8027 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 (DEFAULT_TABLE table=test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.607345 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77ec9143734bfd852f3dcd2a417f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.607352 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77ec9143734bfd852f3dcd2a417f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.608539 8424 ts_tablet_manager.cc:1918] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:10.608636 8424 log.cc:1196] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/b0b42889682848d8a61de8f4d6e24f2d
I20260924 17:39:10.608752 8403 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Bootstrap starting.
I20260924 17:39:10.608934 8424 ts_tablet_manager.cc:1939] T b0b42889682848d8a61de8f4d6e24f2d P 022951de34a141a9a170f647e8941675: Deleting consensus metadata
I20260924 17:39:10.608968 8401 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Bootstrap starting.
I20260924 17:39:10.609156 8403 tablet_bootstrap.cc:654] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.609397 8401 tablet_bootstrap.cc:654] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.609791 7874 catalog_manager.cc:4841] TS 022951de34a141a9a170f647e8941675 (127.0.98.2:38623): tablet b0b42889682848d8a61de8f4d6e24f2d (table test-table [id=81de98e014154e6f9d5b8c2a877451da]) successfully deleted
I20260924 17:39:10.609900 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 (DEFAULT_TABLE table=test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.609989 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77ec9143734bfd852f3dcd2a417f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.610176 8403 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: No bootstrap required, opened a new log
I20260924 17:39:10.610208 8403 ts_tablet_manager.cc:1397] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.610353 8403 raft_consensus.cc:348] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.610407 8403 raft_consensus.cc:374] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.610420 8403 raft_consensus.cc:729] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.610484 8403 consensus_queue.cc:260] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.610592 8403 ts_tablet_manager.cc:1428] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:10.611145 8402 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Bootstrap starting.
I20260924 17:39:10.611488 8402 tablet_bootstrap.cc:654] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.612095 8402 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: No bootstrap required, opened a new log
I20260924 17:39:10.612138 8402 ts_tablet_manager.cc:1397] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.612288 8402 raft_consensus.cc:348] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.612342 8402 raft_consensus.cc:374] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.612356 8402 raft_consensus.cc:729] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022951de34a141a9a170f647e8941675, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.612450 8402 consensus_queue.cc:260] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.612573 8402 ts_tablet_manager.cc:1428] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:10.612721 8426 tablet_replica.cc:330] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: stopping tablet replica
I20260924 17:39:10.612849 8426 raft_consensus.cc:2229] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:10.612975 8426 raft_consensus.cc:2258] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.613447 8426 ts_tablet_manager.cc:1905] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.615181 8425 tablet_replica.cc:330] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: stopping tablet replica
I20260924 17:39:10.615310 8425 raft_consensus.cc:2229] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:10.615360 8426 ts_tablet_manager.cc:1918] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:10.615424 8426 log.cc:1196] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/b0b42889682848d8a61de8f4d6e24f2d
I20260924 17:39:10.615459 8425 raft_consensus.cc:2258] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.615713 8426 ts_tablet_manager.cc:1939] T b0b42889682848d8a61de8f4d6e24f2d P 0e5681b9e18e415ca9778a4ecfd98601: Deleting consensus metadata
I20260924 17:39:10.615926 8401 tablet_bootstrap.cc:492] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: No bootstrap required, opened a new log
I20260924 17:39:10.615971 8401 ts_tablet_manager.cc:1397] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 17:39:10.616112 8401 raft_consensus.cc:348] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.616142 7875 catalog_manager.cc:4841] TS 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163): tablet b0b42889682848d8a61de8f4d6e24f2d (table test-table [id=81de98e014154e6f9d5b8c2a877451da]) successfully deleted
I20260924 17:39:10.616178 8401 raft_consensus.cc:374] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.616184 8425 ts_tablet_manager.cc:1905] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.616221 8401 raft_consensus.cc:729] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.616317 8401 consensus_queue.cc:260] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.616442 8401 ts_tablet_manager.cc:1428] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.617801 8425 ts_tablet_manager.cc:1918] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:10.617859 8425 log.cc:1196] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/b0b42889682848d8a61de8f4d6e24f2d
I20260924 17:39:10.618094 8425 ts_tablet_manager.cc:1939] T b0b42889682848d8a61de8f4d6e24f2d P 840fc1de614d444b82ea9c032d4ab647: Deleting consensus metadata
I20260924 17:39:10.618829 7877 catalog_manager.cc:4841] TS 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207): tablet b0b42889682848d8a61de8f4d6e24f2d (table test-table [id=81de98e014154e6f9d5b8c2a877451da]) successfully deleted
I20260924 17:39:10.751179 8409 raft_consensus.cc:482] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:10.751303 8409 raft_consensus.cc:504] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.751721 8409 leader_election.cc:288] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022951de34a141a9a170f647e8941675 (127.0.98.2:38623), 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207)
I20260924 17:39:10.755694 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.751998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41959 (local address 127.0.98.2:38623)
0924 17:39:10.752058 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.752060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.752405 (+ 345us) server_negotiation.cc:407] Connection header received
0924 17:39:10.752445 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.752446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.752456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.752540 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.752723 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.753410 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.754722 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.755357 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.755420 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.755429 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.755581 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.755624 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:39:10.755694 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.752075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43597 (local address 127.0.98.3:41207)
0924 17:39:10.752137 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.752139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.752150 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:10.752189 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.752190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.752199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.752246 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:10.752448 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.753225 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.754723 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.755294 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.755387 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.755400 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.755581 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.755622 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:39:10.755738 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.751922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.98.1:43597)
0924 17:39:10.752056 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.752067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.752124 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.752247 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.752248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.752249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.752334 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.752337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.753307 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.753308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.754610 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.754614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.755374 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.755375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.755487 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.755492 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.755558 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.755612 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":8,"thread_start_us":51,"threads_started":1}
I20260924 17:39:10.755738 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.751922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.98.1:41959)
0924 17:39:10.752291 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.752302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.752355 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.752547 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.752548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.752549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:10.752641 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.752642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.753496 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.753497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.754614 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.754615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.755448 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.755449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.755508 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.755511 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.755565 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.755612 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":36,"thread_start_us":63,"threads_started":1}
I20260924 17:39:10.756073 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77ec9143734bfd852f3dcd2a417f25" candidate_uuid: "0e5681b9e18e415ca9778a4ecfd98601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647" is_pre_election: true
I20260924 17:39:10.756079 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77ec9143734bfd852f3dcd2a417f25" candidate_uuid: "0e5681b9e18e415ca9778a4ecfd98601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675" is_pre_election: true
I20260924 17:39:10.756175 8327 raft_consensus.cc:2454] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5681b9e18e415ca9778a4ecfd98601 in term 0.
I20260924 17:39:10.756222 8187 raft_consensus.cc:2454] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5681b9e18e415ca9778a4ecfd98601 in term 0.
I20260924 17:39:10.756384 7981 leader_election.cc:302] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5681b9e18e415ca9778a4ecfd98601, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:10.756541 8409 raft_consensus.cc:2781] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:10.756585 8409 raft_consensus.cc:482] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:10.756616 8409 raft_consensus.cc:3037] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.757347 8409 raft_consensus.cc:504] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.757529 8409 leader_election.cc:288] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [CANDIDATE]: Term 1 election: Requested vote from peers 022951de34a141a9a170f647e8941675 (127.0.98.2:38623), 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207)
I20260924 17:39:10.757688 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77ec9143734bfd852f3dcd2a417f25" candidate_uuid: "0e5681b9e18e415ca9778a4ecfd98601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675"
I20260924 17:39:10.757795 8187 raft_consensus.cc:3037] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.757755 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77ec9143734bfd852f3dcd2a417f25" candidate_uuid: "0e5681b9e18e415ca9778a4ecfd98601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647"
I20260924 17:39:10.757860 8327 raft_consensus.cc:3037] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.758463 8327 raft_consensus.cc:2454] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5681b9e18e415ca9778a4ecfd98601 in term 1.
I20260924 17:39:10.758544 8187 raft_consensus.cc:2454] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5681b9e18e415ca9778a4ecfd98601 in term 1.
I20260924 17:39:10.758649 7981 leader_election.cc:302] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5681b9e18e415ca9778a4ecfd98601, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:10.758756 8409 raft_consensus.cc:2781] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:10.758900 8409 raft_consensus.cc:686] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Running, Role: LEADER
I20260924 17:39:10.758958 8409 consensus_queue.cc:237] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.759826 7888 catalog_manager.cc:5495] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0e5681b9e18e415ca9778a4ecfd98601" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:10.818351 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.811900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.98.62:39407)
0924 17:39:10.812008 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.812009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.812092 (+ 83us) server_negotiation.cc:407] Connection header received
0924 17:39:10.812155 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.812155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.812164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.812204 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.812693 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.813183 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.813869 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.814043 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.814096 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.816184 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.816185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.816185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.816192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.817326 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.817695 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.817696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.817696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.817750 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.817921 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.817922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.817922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.818015 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.818062 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.818220 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.818285 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 17:39:10.818507 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.811842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39407 (local address 127.0.0.1:58464)
0924 17:39:10.812021 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.812031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.812097 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.812225 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.812226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.812555 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:10.812605 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.812607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.813253 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.813254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.813798 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.813800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.814148 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.814149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.814199 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.814655 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:10.814660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:10.816117 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:10.817392 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.817398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.817400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.817624 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.817810 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:10.817810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:10.817810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:10.817864 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:10.818087 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:10.818093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:10.818136 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.818203 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.818339 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":45,"thread_start_us":73,"threads_started":1}
I20260924 17:39:10.820406 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58464:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:39:10.820588 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:10.821996 8167 tablet_service.cc:1467] Processing CreateTablet for tablet c51f015c3eec4017a3c10bbc872b7458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ed1327a9c124ae885eb2000759f2310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:39:10.822017 8307 tablet_service.cc:1467] Processing CreateTablet for tablet c51f015c3eec4017a3c10bbc872b7458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ed1327a9c124ae885eb2000759f2310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:39:10.822048 8027 tablet_service.cc:1467] Processing CreateTablet for tablet c51f015c3eec4017a3c10bbc872b7458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ed1327a9c124ae885eb2000759f2310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:39:10.822166 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51f015c3eec4017a3c10bbc872b7458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.822166 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51f015c3eec4017a3c10bbc872b7458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.822199 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51f015c3eec4017a3c10bbc872b7458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.823459 8403 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: Bootstrap starting.
I20260924 17:39:10.823658 8401 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: Bootstrap starting.
I20260924 17:39:10.823661 8402 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: Bootstrap starting.
I20260924 17:39:10.823891 8403 tablet_bootstrap.cc:654] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.824007 8402 tablet_bootstrap.cc:654] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.824049 8401 tablet_bootstrap.cc:654] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.824610 8403 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: No bootstrap required, opened a new log
I20260924 17:39:10.824613 8402 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: No bootstrap required, opened a new log
I20260924 17:39:10.824666 8403 ts_tablet_manager.cc:1397] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.824666 8402 ts_tablet_manager.cc:1397] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.824692 8401 tablet_bootstrap.cc:492] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: No bootstrap required, opened a new log
I20260924 17:39:10.824743 8401 ts_tablet_manager.cc:1397] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:10.824812 8403 raft_consensus.cc:348] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.824788 8402 raft_consensus.cc:348] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.824885 8402 raft_consensus.cc:374] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.824896 8403 raft_consensus.cc:374] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.824923 8403 raft_consensus.cc:729] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.824928 8402 raft_consensus.cc:729] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022951de34a141a9a170f647e8941675, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.824922 8401 raft_consensus.cc:348] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.824970 8403 consensus_queue.cc:260] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.824972 8402 consensus_queue.cc:260] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.825002 8401 raft_consensus.cc:374] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.825032 8401 raft_consensus.cc:729] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.825058 8402 ts_tablet_manager.cc:1428] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:10.825057 8403 ts_tablet_manager.cc:1428] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.825074 8401 consensus_queue.cc:260] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.825181 8401 ts_tablet_manager.cc:1428] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.825409 8409 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.825502 8409 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.825826 8411 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.825933 8411 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.826131 8408 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.826227 8408 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.891141 8411 raft_consensus.cc:482] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:10.891256 8411 raft_consensus.cc:504] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.891441 8411 leader_election.cc:288] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207)
I20260924 17:39:10.891671 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51f015c3eec4017a3c10bbc872b7458" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647" is_pre_election: true
I20260924 17:39:10.891659 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51f015c3eec4017a3c10bbc872b7458" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601" is_pre_election: true
I20260924 17:39:10.891772 8327 raft_consensus.cc:2454] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 0.
I20260924 17:39:10.891772 8047 raft_consensus.cc:2454] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 0.
I20260924 17:39:10.891906 8121 leader_election.cc:302] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022951de34a141a9a170f647e8941675, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:10.892041 8411 raft_consensus.cc:2781] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:10.892093 8411 raft_consensus.cc:482] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:10.892138 8411 raft_consensus.cc:3037] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.892889 8411 raft_consensus.cc:504] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.893028 8411 leader_election.cc:288] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207)
I20260924 17:39:10.893169 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51f015c3eec4017a3c10bbc872b7458" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601"
I20260924 17:39:10.893196 8327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51f015c3eec4017a3c10bbc872b7458" candidate_uuid: "022951de34a141a9a170f647e8941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840fc1de614d444b82ea9c032d4ab647"
I20260924 17:39:10.893265 8047 raft_consensus.cc:3037] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.893278 8327 raft_consensus.cc:3037] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:10.893985 8047 raft_consensus.cc:2454] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 1.
I20260924 17:39:10.893985 8327 raft_consensus.cc:2454] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022951de34a141a9a170f647e8941675 in term 1.
I20260924 17:39:10.894161 8119 leader_election.cc:302] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022951de34a141a9a170f647e8941675, 0e5681b9e18e415ca9778a4ecfd98601; no voters:
I20260924 17:39:10.894286 8411 raft_consensus.cc:2781] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:10.894335 8411 raft_consensus.cc:686] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [term 1 LEADER]: Becoming Leader. State: Replica: 022951de34a141a9a170f647e8941675, State: Running, Role: LEADER
I20260924 17:39:10.894382 8411 consensus_queue.cc:237] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.894642 8419 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: TxnStatusTablet state changed. Reason: New leader 022951de34a141a9a170f647e8941675. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.894719 8419 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:39:10.894867 8444 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:39:10.894980 8444 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:39:10.895058 7888 catalog_manager.cc:5495] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022951de34a141a9a170f647e8941675 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: UNKNOWN } } }
I20260924 17:39:10.900883 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.898511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.0.98.2:38623)
0924 17:39:10.898558 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.898560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.898570 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:10.898650 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.898651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.898665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.898723 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:10.898826 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.899377 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.900145 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.900349 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.900432 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.900615 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.900651 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.900720 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.900785 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.900808 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:39:10.900897 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.898463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.0.1:35674)
0924 17:39:10.898519 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.898531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.898590 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.898728 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.898729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.898730 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:39:10.898778 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.898780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.899460 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.899461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.900042 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.900044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.900432 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.900433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.900489 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.900515 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.900722 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.900731 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.900773 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.900826 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 17:39:10.901913 8047 raft_consensus.cc:1261] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Refusing update from remote peer 022951de34a141a9a170f647e8941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.901898 8327 raft_consensus.cc:1261] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Refusing update from remote peer 022951de34a141a9a170f647e8941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.902136 8419 consensus_queue.cc:1035] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.902282 8419 consensus_queue.cc:1035] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.902642 8408 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: TxnStatusTablet state changed. Reason: New leader 022951de34a141a9a170f647e8941675. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.902737 8408 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.903249 8409 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: TxnStatusTablet state changed. Reason: New leader 022951de34a141a9a170f647e8941675. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.903347 8409 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.903414 8411 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.903501 8411 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:39:10.903893 8407 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.903961 8407 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 022951de34a141a9a170f647e8941675: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:39:10.904247 8409 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.904340 8409 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 0e5681b9e18e415ca9778a4ecfd98601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.904767 8408 tablet_replica.cc:439] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022951de34a141a9a170f647e8941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } }
I20260924 17:39:10.904847 8408 tablet_replica.cc:442] T c51f015c3eec4017a3c10bbc872b7458 P 840fc1de614d444b82ea9c032d4ab647: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:39:10.904963 8415 mvcc.cc:205] Tried to move back new op lower bound from 7332952272492539904 to 7332952272463552512. Current Snapshot: MvccSnapshot[applied={T|T < 7332952272492539904 or (T in {7332952272492539904})}]
I20260924 17:39:10.907806 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.905277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.0.98.1:36163)
0924 17:39:10.905313 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.905316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.905323 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:39:10.905356 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.905357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.905377 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.905442 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:10.905623 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.906229 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.906947 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.907235 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.907308 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.907468 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.907587 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.907671 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.907711 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.907732 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:10.912528 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.910200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36087 (local address 127.0.98.2:38623)
0924 17:39:10.910264 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.910266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.910275 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:10.910346 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.910347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.910359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.910411 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:10.910548 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.911009 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.911835 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.912039 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.912125 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.912291 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.912323 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.912373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.912438 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.912458 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:39:10.912551 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.910070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.98.1:36087)
0924 17:39:10.910207 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.910232 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.910287 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.910416 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.910417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.910419 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:39:10.910476 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.910478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.911141 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.911142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.911736 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.911738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.912126 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.912127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.912189 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.912201 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.912382 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.912383 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.912433 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.912473 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260924 17:39:10.915340 8327 raft_consensus.cc:1261] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Refusing update from remote peer 0e5681b9e18e415ca9778a4ecfd98601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.915369 8187 raft_consensus.cc:1261] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Refusing update from remote peer 0e5681b9e18e415ca9778a4ecfd98601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:10.915611 8409 consensus_queue.cc:1035] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.915776 8432 consensus_queue.cc:1035] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:10.924288 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.921645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36163 (local address 127.0.98.2:37331)
0924 17:39:10.921823 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:10.921832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:10.921905 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:10.922026 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:10.922026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:10.922028 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:39:10.922074 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.922076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.922668 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.922669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.923485 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:10.923488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.923898 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.923898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:10.923936 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.923948 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.924130 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.924132 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:39:10.924171 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 17:39:10.924217 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260924 17:39:10.924278 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.921705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37331 (local address 127.0.98.1:36163)
0924 17:39:10.921744 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.921746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.921889 (+ 143us) server_negotiation.cc:407] Connection header received
0924 17:39:10.921961 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.921962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.921972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.922023 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:10.922123 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.922589 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.923589 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.923821 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.923853 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.924029 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:10.924070 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:10.924122 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.924184 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.924203 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:10.932017 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:10.932135 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:10.932753 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Sending DeleteTablet for 3 replicas of tablet 2c77ec9143734bfd852f3dcd2a417f25
I20260924 17:39:10.933096 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260924 17:39:10.933203 8425 tablet_replica.cc:330] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: stopping tablet replica
I20260924 17:39:10.933264 8425 raft_consensus.cc:2229] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:10.933260 8027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 17:39:10.933310 8425 raft_consensus.cc:2258] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.933336 8426 tablet_replica.cc:330] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: stopping tablet replica
I20260924 17:39:10.933385 8426 raft_consensus.cc:2229] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:10.933434 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c77ec9143734bfd852f3dcd2a417f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35658
I20260924 17:39:10.933485 8426 raft_consensus.cc:2258] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.933543 8424 tablet_replica.cc:330] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: stopping tablet replica
I20260924 17:39:10.933616 8424 raft_consensus.cc:2229] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:10.933645 8425 ts_tablet_manager.cc:1905] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.933669 8424 raft_consensus.cc:2258] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:10.933964 8424 ts_tablet_manager.cc:1905] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.935124 8425 ts_tablet_manager.cc:1918] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:39:10.935179 8425 log.cc:1196] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/2c77ec9143734bfd852f3dcd2a417f25
I20260924 17:39:10.935421 8425 ts_tablet_manager.cc:1939] T 2c77ec9143734bfd852f3dcd2a417f25 P 840fc1de614d444b82ea9c032d4ab647: Deleting consensus metadata
I20260924 17:39:10.935560 8424 ts_tablet_manager.cc:1918] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:39:10.935611 8424 log.cc:1196] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/2c77ec9143734bfd852f3dcd2a417f25
I20260924 17:39:10.935729 7877 catalog_manager.cc:4841] TS 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207): tablet 2c77ec9143734bfd852f3dcd2a417f25 (table test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]) successfully deleted
I20260924 17:39:10.935817 8424 ts_tablet_manager.cc:1939] T 2c77ec9143734bfd852f3dcd2a417f25 P 022951de34a141a9a170f647e8941675: Deleting consensus metadata
I20260924 17:39:10.936165 7874 catalog_manager.cc:4841] TS 022951de34a141a9a170f647e8941675 (127.0.98.2:38623): tablet 2c77ec9143734bfd852f3dcd2a417f25 (table test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]) successfully deleted
I20260924 17:39:10.936321 8426 ts_tablet_manager.cc:1905] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:10.937798 8426 ts_tablet_manager.cc:1918] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:39:10.937847 8426 log.cc:1196] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/2c77ec9143734bfd852f3dcd2a417f25
I20260924 17:39:10.938046 8426 ts_tablet_manager.cc:1939] T 2c77ec9143734bfd852f3dcd2a417f25 P 0e5681b9e18e415ca9778a4ecfd98601: Deleting consensus metadata
I20260924 17:39:10.938437 7875 catalog_manager.cc:4841] TS 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163): tablet 2c77ec9143734bfd852f3dcd2a417f25 (table test-table [id=ab21a0abd2b54aad943aa995ffb01ebb]) successfully deleted
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:39:10.968684 7940 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:10.961519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.0.98.62:39407)
0924 17:39:10.961569 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.961571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.961737 (+ 166us) server_negotiation.cc:407] Connection header received
0924 17:39:10.961799 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.961800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.961815 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.961881 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.962689 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.963579 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.964682 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.964998 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.965071 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.968617 (+ 3546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58474: BlockingRecv error: recv got EOF from 127.0.0.1:58474 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 17:39:10 dist-test-slave-d0pf krb5kdc[7845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 17:39:10.978401 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:10.969441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.0.98.62:39407)
0924 17:39:10.969491 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:10.969493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:10.969697 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:39:10.969805 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:10.969806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:10.969819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:10.969886 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:10.970475 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.971350 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.972342 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:10.972655 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:10.972728 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:10.975425 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:10.975426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:10.975427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:10.975435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:10.977155 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.977569 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.977570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.977570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.977637 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:10.977852 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:10.977853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:10.977853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:10.978001 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:10.978116 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:10.978223 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:39:10.978303 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:39:10.980464 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:10.980643 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:10.982084 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb29abf8e449a9871005a5f65a55ce (DEFAULT_TABLE table=test-table [id=b9cc45db623942a9a1be63c1c710dfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.982170 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb29abf8e449a9871005a5f65a55ce (DEFAULT_TABLE table=test-table [id=b9cc45db623942a9a1be63c1c710dfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.982221 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb29abf8e449a9871005a5f65a55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.982295 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb29abf8e449a9871005a5f65a55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.983172 8027 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb29abf8e449a9871005a5f65a55ce (DEFAULT_TABLE table=test-table [id=b9cc45db623942a9a1be63c1c710dfec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:10.983295 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb29abf8e449a9871005a5f65a55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:10.983614 8402 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Bootstrap starting.
I20260924 17:39:10.984011 8402 tablet_bootstrap.cc:654] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.984608 8403 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Bootstrap starting.
I20260924 17:39:10.985018 8403 tablet_bootstrap.cc:654] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.986338 8401 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Bootstrap starting.
I20260924 17:39:10.986590 8403 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: No bootstrap required, opened a new log
I20260924 17:39:10.986639 8403 ts_tablet_manager.cc:1397] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:39:10.986768 8401 tablet_bootstrap.cc:654] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:10.986778 8403 raft_consensus.cc:348] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.986847 8403 raft_consensus.cc:374] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.986869 8403 raft_consensus.cc:729] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.986912 8403 consensus_queue.cc:260] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.986991 8403 ts_tablet_manager.cc:1428] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:10.987917 8401 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: No bootstrap required, opened a new log
I20260924 17:39:10.987918 8402 tablet_bootstrap.cc:492] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: No bootstrap required, opened a new log
I20260924 17:39:10.987967 8401 ts_tablet_manager.cc:1397] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:39:10.987970 8402 ts_tablet_manager.cc:1397] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 17:39:10.988112 8401 raft_consensus.cc:348] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.988132 8402 raft_consensus.cc:348] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.988178 8401 raft_consensus.cc:374] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.988199 8401 raft_consensus.cc:729] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.988200 8402 raft_consensus.cc:374] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:10.988219 8402 raft_consensus.cc:729] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022951de34a141a9a170f647e8941675, State: Initialized, Role: FOLLOWER
I20260924 17:39:10.988248 8401 consensus_queue.cc:260] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.988260 8402 consensus_queue.cc:260] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:10.988327 8401 ts_tablet_manager.cc:1428] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:10.988344 8402 ts_tablet_manager.cc:1428] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:11.026845 8408 raft_consensus.cc:482] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:11.026974 8408 raft_consensus.cc:504] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.027356 8408 leader_election.cc:288] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 022951de34a141a9a170f647e8941675 (127.0.98.2:38623)
I20260924 17:39:11.031996 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.027974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38545 (local address 127.0.98.2:38623)
0924 17:39:11.028020 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:11.028022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:11.028115 (+ 93us) server_negotiation.cc:407] Connection header received
0924 17:39:11.028212 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:11.028213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:11.028222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:11.028251 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:11.028470 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.029151 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.030681 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.031252 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.031303 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.031315 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:11.031880 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 17:39:11.031923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:39:11.031994 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.027834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49465 (local address 127.0.98.1:36163)
0924 17:39:11.027881 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:11.027883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:11.027931 (+ 48us) server_negotiation.cc:407] Connection header received
0924 17:39:11.027961 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:11.027961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:11.027970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:11.028020 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:11.028230 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.028944 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.030954 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.031523 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.031574 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.031583 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:11.031870 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 17:39:11.031909 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:39:11.032254 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.027546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36163 (local address 127.0.98.3:49465)
0924 17:39:11.027806 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:11.027819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:11.027879 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:11.028071 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:11.028073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:11.028075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:11.028152 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.028154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.029055 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.029056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.030846 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.030849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.031705 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.031706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.031775 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.031780 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:39:11.032124 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 17:39:11.032169 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
I20260924 17:39:11.032483 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb29abf8e449a9871005a5f65a55ce" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601" is_pre_election: true
I20260924 17:39:11.032596 8047 raft_consensus.cc:2454] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 0.
I20260924 17:39:11.032799 8259 leader_election.cc:302] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5681b9e18e415ca9778a4ecfd98601, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:11.032792 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.027654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.98.3:38545)
0924 17:39:11.028058 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:11.028068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:11.028137 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:11.028316 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:11.028316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:11.028318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:11.028392 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.028394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.029276 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.029277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.030577 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.030581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.031701 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.031702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.031788 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.031792 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:11.032572 (+ 780us) client_negotiation.cc:240] Negotiation successful
0924 17:39:11.032712 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":64,"threads_started":1}
I20260924 17:39:11.032951 8408 raft_consensus.cc:2781] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:11.032995 8408 raft_consensus.cc:482] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:11.033015 8408 raft_consensus.cc:3037] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.033434 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb29abf8e449a9871005a5f65a55ce" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675" is_pre_election: true
I20260924 17:39:11.033540 8187 raft_consensus.cc:2454] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 0.
I20260924 17:39:11.033910 8408 raft_consensus.cc:504] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.034056 8408 leader_election.cc:288] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 022951de34a141a9a170f647e8941675 (127.0.98.2:38623)
I20260924 17:39:11.034238 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb29abf8e449a9871005a5f65a55ce" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675"
I20260924 17:39:11.034313 8187 raft_consensus.cc:3037] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.034325 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb29abf8e449a9871005a5f65a55ce" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601"
I20260924 17:39:11.034400 8047 raft_consensus.cc:3037] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.035142 8187 raft_consensus.cc:2454] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 1.
I20260924 17:39:11.035260 8047 raft_consensus.cc:2454] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 1.
I20260924 17:39:11.035306 8258 leader_election.cc:302] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022951de34a141a9a170f647e8941675, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:11.035382 8408 raft_consensus.cc:2781] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:11.035482 8408 raft_consensus.cc:686] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 1 LEADER]: Becoming Leader. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Running, Role: LEADER
I20260924 17:39:11.035547 8408 consensus_queue.cc:237] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.036247 7890 catalog_manager.cc:5495] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 reported cstate change: term changed from 0 to 1, leader changed from <none> to 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "840fc1de614d444b82ea9c032d4ab647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: HEALTHY } } }
I20260924 17:39:11.048647 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.045336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.98.3:41207)
0924 17:39:11.045380 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:11.045383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:11.045391 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:11.045437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:11.045438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:11.045458 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:11.045538 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:11.045719 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.046445 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.047579 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.047839 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.047903 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.048109 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:11.048233 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:11.048301 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:11.048547 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 17:39:11.048567 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:39:11.049834 8187 raft_consensus.cc:1261] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Refusing update from remote peer 840fc1de614d444b82ea9c032d4ab647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:11.050077 8476 consensus_queue.cc:1035] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:11.050251 8047 raft_consensus.cc:1261] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Refusing update from remote peer 840fc1de614d444b82ea9c032d4ab647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:11.050726 8408 consensus_queue.cc:1035] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:11.053992 7889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:11.054085 7889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:11.054737 7889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Sending DeleteTablet for 3 replicas of tablet 1beb29abf8e449a9871005a5f65a55ce
I20260924 17:39:11.054962 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb29abf8e449a9871005a5f65a55ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260924 17:39:11.055079 8425 tablet_replica.cc:330] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: stopping tablet replica
I20260924 17:39:11.055130 8425 raft_consensus.cc:2229] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:11.055214 8425 raft_consensus.cc:2258] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.055429 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb29abf8e449a9871005a5f65a55ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35658
I20260924 17:39:11.055531 8424 tablet_replica.cc:330] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: stopping tablet replica
I20260924 17:39:11.055589 8424 raft_consensus.cc:2229] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:11.055636 8424 raft_consensus.cc:2258] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.055863 7889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:11.056025 7889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:11.056182 8027 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb29abf8e449a9871005a5f65a55ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 17:39:11.056263 8426 tablet_replica.cc:330] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: stopping tablet replica
I20260924 17:39:11.056313 8426 raft_consensus.cc:2229] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:11.056356 8426 raft_consensus.cc:2258] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.056730 8424 ts_tablet_manager.cc:1905] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:11.056764 8425 ts_tablet_manager.cc:1905] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:11.058198 8425 ts_tablet_manager.cc:1918] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:11.058261 8425 log.cc:1196] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/1beb29abf8e449a9871005a5f65a55ce
I20260924 17:39:11.058395 8424 ts_tablet_manager.cc:1918] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:11.058456 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 05d9910931734283972f5ec1677cbd80 (DEFAULT_TABLE table=test-table [id=d7da41a69fa746059b33a70d96faa8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:11.058502 8424 log.cc:1196] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/1beb29abf8e449a9871005a5f65a55ce
I20260924 17:39:11.058504 8425 ts_tablet_manager.cc:1939] T 1beb29abf8e449a9871005a5f65a55ce P 840fc1de614d444b82ea9c032d4ab647: Deleting consensus metadata
I20260924 17:39:11.058544 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d9910931734283972f5ec1677cbd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:11.058745 8424 ts_tablet_manager.cc:1939] T 1beb29abf8e449a9871005a5f65a55ce P 022951de34a141a9a170f647e8941675: Deleting consensus metadata
I20260924 17:39:11.058794 8426 ts_tablet_manager.cc:1905] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:11.059327 7874 catalog_manager.cc:4841] TS 022951de34a141a9a170f647e8941675 (127.0.98.2:38623): tablet 1beb29abf8e449a9871005a5f65a55ce (table test-table [id=b9cc45db623942a9a1be63c1c710dfec]) successfully deleted
I20260924 17:39:11.059492 7877 catalog_manager.cc:4841] TS 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207): tablet 1beb29abf8e449a9871005a5f65a55ce (table test-table [id=b9cc45db623942a9a1be63c1c710dfec]) successfully deleted
I20260924 17:39:11.059641 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 05d9910931734283972f5ec1677cbd80 (DEFAULT_TABLE table=test-table [id=d7da41a69fa746059b33a70d96faa8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:11.059731 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d9910931734283972f5ec1677cbd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:11.059873 8402 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Bootstrap starting.
I20260924 17:39:11.060143 8426 ts_tablet_manager.cc:1918] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:11.060194 8426 log.cc:1196] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/1beb29abf8e449a9871005a5f65a55ce
I20260924 17:39:11.060189 8402 tablet_bootstrap.cc:654] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:11.060386 8426 ts_tablet_manager.cc:1939] T 1beb29abf8e449a9871005a5f65a55ce P 0e5681b9e18e415ca9778a4ecfd98601: Deleting consensus metadata
I20260924 17:39:11.060570 8027 tablet_service.cc:1467] Processing CreateTablet for tablet 05d9910931734283972f5ec1677cbd80 (DEFAULT_TABLE table=test-table [id=d7da41a69fa746059b33a70d96faa8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:11.060660 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d9910931734283972f5ec1677cbd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:11.060753 7875 catalog_manager.cc:4841] TS 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163): tablet 1beb29abf8e449a9871005a5f65a55ce (table test-table [id=b9cc45db623942a9a1be63c1c710dfec]) successfully deleted
I20260924 17:39:11.061004 8401 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Bootstrap starting.
I20260924 17:39:11.061347 8401 tablet_bootstrap.cc:654] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:11.061795 8403 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: Bootstrap starting.
I20260924 17:39:11.062182 8403 tablet_bootstrap.cc:654] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:11.062721 8403 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: No bootstrap required, opened a new log
I20260924 17:39:11.062768 8403 ts_tablet_manager.cc:1397] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:11.062806 8401 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: No bootstrap required, opened a new log
I20260924 17:39:11.062846 8401 ts_tablet_manager.cc:1397] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:39:11.062932 8403 raft_consensus.cc:348] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.062971 8401 raft_consensus.cc:348] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.063009 8403 raft_consensus.cc:374] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:11.063054 8401 raft_consensus.cc:374] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:11.063066 8403 raft_consensus.cc:729] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5681b9e18e415ca9778a4ecfd98601, State: Initialized, Role: FOLLOWER
I20260924 17:39:11.063081 8401 raft_consensus.cc:729] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Initialized, Role: FOLLOWER
I20260924 17:39:11.063112 8403 consensus_queue.cc:260] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.063136 8401 consensus_queue.cc:260] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.063211 8403 ts_tablet_manager.cc:1428] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:11.063221 8401 ts_tablet_manager.cc:1428] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:11.064230 8402 tablet_bootstrap.cc:492] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: No bootstrap required, opened a new log
I20260924 17:39:11.064294 8402 ts_tablet_manager.cc:1397] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 17:39:11.064421 8402 raft_consensus.cc:348] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.064499 8402 raft_consensus.cc:374] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:11.064527 8402 raft_consensus.cc:729] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022951de34a141a9a170f647e8941675, State: Initialized, Role: FOLLOWER
I20260924 17:39:11.064576 8402 consensus_queue.cc:260] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.064683 8402 ts_tablet_manager.cc:1428] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:11.303107 8476 raft_consensus.cc:482] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:11.303239 8476 raft_consensus.cc:504] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.303421 8476 leader_election.cc:288] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 022951de34a141a9a170f647e8941675 (127.0.98.2:38623)
I20260924 17:39:11.303659 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d9910931734283972f5ec1677cbd80" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601" is_pre_election: true
I20260924 17:39:11.303776 8047 raft_consensus.cc:2454] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 0.
I20260924 17:39:11.303733 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d9910931734283972f5ec1677cbd80" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675" is_pre_election: true
I20260924 17:39:11.303848 8187 raft_consensus.cc:2454] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 0.
I20260924 17:39:11.303959 8259 leader_election.cc:302] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5681b9e18e415ca9778a4ecfd98601, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:11.304070 8476 raft_consensus.cc:2781] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:11.304121 8476 raft_consensus.cc:482] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:11.304149 8476 raft_consensus.cc:3037] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.304926 8476 raft_consensus.cc:504] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.305078 8476 leader_election.cc:288] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163), 022951de34a141a9a170f647e8941675 (127.0.98.2:38623)
I20260924 17:39:11.305249 8047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d9910931734283972f5ec1677cbd80" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5681b9e18e415ca9778a4ecfd98601"
I20260924 17:39:11.305255 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d9910931734283972f5ec1677cbd80" candidate_uuid: "840fc1de614d444b82ea9c032d4ab647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022951de34a141a9a170f647e8941675"
I20260924 17:39:11.305330 8047 raft_consensus.cc:3037] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.305349 8187 raft_consensus.cc:3037] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:11.305960 8047 raft_consensus.cc:2454] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 1.
I20260924 17:39:11.306056 8187 raft_consensus.cc:2454] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840fc1de614d444b82ea9c032d4ab647 in term 1.
I20260924 17:39:11.306149 8259 leader_election.cc:302] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5681b9e18e415ca9778a4ecfd98601, 840fc1de614d444b82ea9c032d4ab647; no voters:
I20260924 17:39:11.306275 8476 raft_consensus.cc:2781] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:11.306353 8476 raft_consensus.cc:686] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 1 LEADER]: Becoming Leader. State: Replica: 840fc1de614d444b82ea9c032d4ab647, State: Running, Role: LEADER
I20260924 17:39:11.306406 8476 consensus_queue.cc:237] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } }
I20260924 17:39:11.307062 7890 catalog_manager.cc:5495] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 reported cstate change: term changed from 0 to 1, leader changed from <none> to 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "840fc1de614d444b82ea9c032d4ab647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "840fc1de614d444b82ea9c032d4ab647" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: HEALTHY } } }
I20260924 17:39:11.350931 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.348659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.98.3:37293)
0924 17:39:11.348819 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:11.348831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:11.348893 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:11.349010 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:11.349011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:11.349013 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:39:11.349057 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.349059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.349621 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.349622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.350206 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.350208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.350568 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.350569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.350614 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.350627 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:39:11.350808 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:39:11.350809 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:11.350839 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 17:39:11.350872 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260924 17:39:11.350929 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.348784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37293 (local address 127.0.98.2:38623)
0924 17:39:11.348835 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:11.348837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:11.348881 (+ 44us) server_negotiation.cc:407] Connection header received
0924 17:39:11.348947 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:11.348948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:11.348956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:11.349001 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:11.349104 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.349570 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.350288 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.350472 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.350551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.350717 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:11.350748 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:11.350798 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:11.350849 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 17:39:11.350867 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:11.352977 8187 raft_consensus.cc:1261] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Refusing update from remote peer 840fc1de614d444b82ea9c032d4ab647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:11.352977 8047 raft_consensus.cc:1261] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Refusing update from remote peer 840fc1de614d444b82ea9c032d4ab647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:11.353195 8476 consensus_queue.cc:1035] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5681b9e18e415ca9778a4ecfd98601" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:11.353291 8481 consensus_queue.cc:1035] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022951de34a141a9a170f647e8941675" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:11.360299 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.357926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39829 (local address 127.0.98.3:41207)
0924 17:39:11.358052 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:11.358053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:11.358059 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:39:11.358074 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:11.358074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:11.358082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:11.358123 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:11.358235 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.358721 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.359585 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.359863 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.359926 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.360067 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:11.360099 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:11.360145 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:11.360209 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 17:39:11.360228 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 17:39:11.360324 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.357897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.98.2:39829)
0924 17:39:11.357941 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:11.357952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:11.357985 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:11.358144 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:11.358145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:11.358146 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:39:11.358191 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.358192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.358783 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.358784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.359487 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:11.359491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:11.359935 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:11.359936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:11.359997 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:11.360005 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:39:11.360144 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:39:11.360145 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:39:11.360207 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:39:11.360252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 17:39:11.369174 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:11.369318 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:11.369988 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79887010157c4c36ac79fd72d702c31a: Sending DeleteTablet for 3 replicas of tablet 05d9910931734283972f5ec1677cbd80
I20260924 17:39:11.370270 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d9910931734283972f5ec1677cbd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35658
I20260924 17:39:11.370266 8027 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d9910931734283972f5ec1677cbd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 17:39:11.370293 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d9910931734283972f5ec1677cbd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260924 17:39:11.370379 8426 tablet_replica.cc:330] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601: stopping tablet replica
I20260924 17:39:11.370393 8425 tablet_replica.cc:330] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: stopping tablet replica
I20260924 17:39:11.370441 8426 raft_consensus.cc:2229] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:11.370460 8425 raft_consensus.cc:2229] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:11.370484 8426 raft_consensus.cc:2258] T 05d9910931734283972f5ec1677cbd80 P 0e5681b9e18e415ca9778a4ecfd98601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.370527 8425 raft_consensus.cc:2258] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.370558 8424 tablet_replica.cc:330] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: stopping tablet replica
I20260924 17:39:11.370638 8424 raft_consensus.cc:2229] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:11.370678 8424 raft_consensus.cc:2258] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:11.370970 8425 ts_tablet_manager.cc:1905] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:11.370982 8424 ts_tablet_manager.cc:1905] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:11.372200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7962
I20260924 17:39:11.373057 8424 ts_tablet_manager.cc:1918] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:39:11.373143 8424 log.cc:1196] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/05d9910931734283972f5ec1677cbd80
I20260924 17:39:11.373373 8424 ts_tablet_manager.cc:1939] T 05d9910931734283972f5ec1677cbd80 P 022951de34a141a9a170f647e8941675: Deleting consensus metadata
I20260924 17:39:11.373749 7874 catalog_manager.cc:4841] TS 022951de34a141a9a170f647e8941675 (127.0.98.2:38623): tablet 05d9910931734283972f5ec1677cbd80 (table test-table [id=d7da41a69fa746059b33a70d96faa8d3]) successfully deleted
I20260924 17:39:11.374958 8425 ts_tablet_manager.cc:1918] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:39:11.375038 8425 log.cc:1196] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/05d9910931734283972f5ec1677cbd80
I20260924 17:39:11.375258 8425 ts_tablet_manager.cc:1939] T 05d9910931734283972f5ec1677cbd80 P 840fc1de614d444b82ea9c032d4ab647: Deleting consensus metadata
I20260924 17:39:11.375613 7877 catalog_manager.cc:4841] TS 840fc1de614d444b82ea9c032d4ab647 (127.0.98.3:41207): tablet 05d9910931734283972f5ec1677cbd80 (table test-table [id=d7da41a69fa746059b33a70d96faa8d3]) successfully deleted
W20260924 17:39:11.379328 7875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36163 (error 108)
I20260924 17:39:11.379535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8102
I20260924 17:39:11.383253 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:11.379715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36163 (local address 127.0.0.1:59140)
0924 17:39:11.383158 (+ 3443us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:11.383213 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3415,"thread_start_us":38,"threads_started":1}
W20260924 17:39:11.383435 7875 catalog_manager.cc:4584] TS 0e5681b9e18e415ca9778a4ecfd98601 (127.0.98.1:36163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05d9910931734283972f5ec1677cbd80: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36163: connect: Connection refused (error 111)
I20260924 17:39:11.385535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8242
I20260924 17:39:11.391780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 7858
2026-09-24T17:39:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2953 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:11 dist-test-slave-d0pf krb5kdc[8496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:11 dist-test-slave-d0pf krb5kdc[8496](info): set up 2 sockets
Sep 24 17:39:11 dist-test-slave-d0pf krb5kdc[8496](info): commencing operation
krb5kdc: starting...
W20260924 17:39:12.435626 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:12 dist-test-slave-d0pf krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:39:12.566174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:12.648051 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:12.648306 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:12.648348 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:12.649545 8509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:12.649616 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:12.649648 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:12.649675 8509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:12.649705 8509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:12.651293 8509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:12.651579 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:12.651826 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:12.656764 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:12.656790 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:12.656831 8509 server_base.cc:1034] running on GCE node
W20260924 17:39:12.656797 8517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:12.657171 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:12.657524 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:12.658763 8509 hybrid_clock.cc:648] HybridClock initialized: now 1790271552658736 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T17:39:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1270 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:12 dist-test-slave-d0pf krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:12 dist-test-slave-d0pf krb5kdc[8522](info): set up 2 sockets
Sep 24 17:39:12 dist-test-slave-d0pf krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260924 17:39:13.711696 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:13 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:39:13.836071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:13.918203 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:13.918435 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:13.918459 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:13.919653 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:13.919697 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:13.919713 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:13.919725 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:13.919737 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:13.921242 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:13.921452 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:13.921628 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:13.926612 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:13.926611 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:13.926647 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:13.926811 8535 server_base.cc:1034] running on GCE node
I20260924 17:39:13.927191 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:13.927568 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:13.928736 8535 hybrid_clock.cc:648] HybridClock initialized: now 1790271553928726 us; error 35 us; skew 500 ppm
Sep 24 17:39:13 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:13.931422 8535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:39:13.931885 8535 webserver.cc:460] Webserver started at http://127.0.98.62:46031/ using document root <none> and password file <none>
I20260924 17:39:13.932123 8535 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:13.932191 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:13.932346 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:13.933437 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "52c0b3050a1c4d918ac0d35fa721a547"
format_stamp: "Formatted at 2026-09-24 17:39:13 on dist-test-slave-d0pf"
server_key: "70111f5c915e0729ce1df0836e34e1ef"
server_key_iv: "752b42837cfcab9b68018c45c0edf97e"
server_key_version: "encryptionkey@0"
I20260924 17:39:13.933809 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "52c0b3050a1c4d918ac0d35fa721a547"
format_stamp: "Formatted at 2026-09-24 17:39:13 on dist-test-slave-d0pf"
server_key: "70111f5c915e0729ce1df0836e34e1ef"
server_key_iv: "752b42837cfcab9b68018c45c0edf97e"
server_key_version: "encryptionkey@0"
I20260924 17:39:13.935276 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 17:39:13.936100 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:13.936299 8535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:13.936412 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "52c0b3050a1c4d918ac0d35fa721a547"
format_stamp: "Formatted at 2026-09-24 17:39:13 on dist-test-slave-d0pf"
server_key: "70111f5c915e0729ce1df0836e34e1ef"
server_key_iv: "752b42837cfcab9b68018c45c0edf97e"
server_key_version: "encryptionkey@0"
I20260924 17:39:13.936492 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:13.949227 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:13.951783 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:13.951921 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:13.955793 8535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260924 17:39:13.955824 8602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260924 17:39:13.956387 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:13.957127 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:13.959499 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Bootstrap starting.
I20260924 17:39:13.960158 8603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:13.960469 8603 log.cc:824] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:13.961337 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: No bootstrap required, opened a new log
I20260924 17:39:13.961375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8535
I20260924 17:39:13.961516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:13.961728 392 external_mini_cluster.cc:1428] Setting key 5a3b3576bb742d03e437daa9441ecbc5
I20260924 17:39:13.963850 8603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:13.964041 8603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:13.964108 8603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b3050a1c4d918ac0d35fa721a547, State: Initialized, Role: FOLLOWER
I20260924 17:39:13.964252 8603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:13.964358 8603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:13.964404 8603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:13.964462 8603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:13.965065 8603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:13.965194 8603 leader_election.cc:302] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c0b3050a1c4d918ac0d35fa721a547; no voters:
I20260924 17:39:13.965412 8603 leader_election.cc:288] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:13.965446 8608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:13.965618 8608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0b3050a1c4d918ac0d35fa721a547, State: Running, Role: LEADER
I20260924 17:39:13.965692 8603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:13.965799 8608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:13.966192 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c0b3050a1c4d918ac0d35fa721a547. Latest consensus state: current_term: 1 leader_uuid: "52c0b3050a1c4d918ac0d35fa721a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:39:13.966486 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:13.966389 8610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c0b3050a1c4d918ac0d35fa721a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:39:13.966923 8610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: This master's current role is: LEADER
W20260924 17:39:13.967742 8624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:39:13.967799 8624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:39:13.967849 8617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:13.968007 8617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:39:13 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:13.972075 8617 catalog_manager.cc:1347] Generated new cluster ID: 1d57fb9f17c34d32a7f8d07621a7f8d5
I20260924 17:39:13.972142 8617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:13.979368 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:13.962309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.0.98.62:45549)
0924 17:39:13.962509 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:13.962519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:13.962554 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:39:13.963840 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:13.963850 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:13.963917 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:13.964096 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:13.964845 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:13.966271 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:13.967454 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:13.967980 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:13.968113 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:13.974155 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:13.974162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:13.974162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:13.974179 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:13.976718 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:13.977749 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:13.977753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:13.977753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:13.977915 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:13.978175 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:13.978176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:13.978176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:13.978589 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:13.978772 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:13.978898 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 17:39:13.979187 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 17:39:13.985560 8617 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:13.986327 8617 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:13.996924 8617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Generated new TSK 0
I20260924 17:39:13.997149 8617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:39:14.028494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:14.118269 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.118530 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.118568 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.119966 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.120052 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.120101 8629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:14.121763 8629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.122062 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.122339 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:14.127525 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.127529 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.127529 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.127696 8629 server_base.cc:1034] running on GCE node
I20260924 17:39:14.128093 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.128363 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.129530 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790271554129511 us; error 39 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.132258 8629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:39:14.132769 8629 webserver.cc:460] Webserver started at http://127.0.98.1:34531/ using document root <none> and password file <none>
I20260924 17:39:14.133019 8629 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.133091 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.133242 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:14.134308 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "9569e8a8b8784b8e92817042cdb0de33"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "53f1cddd19611b675cb64620f5932d78"
server_key_iv: "232e4a062a889de3e14d59f0e7242b4d"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.134675 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9569e8a8b8784b8e92817042cdb0de33"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "53f1cddd19611b675cb64620f5932d78"
server_key_iv: "232e4a062a889de3e14d59f0e7242b4d"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.136058 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:39:14.136842 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.137063 8629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.137172 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "9569e8a8b8784b8e92817042cdb0de33"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "53f1cddd19611b675cb64620f5932d78"
server_key_iv: "232e4a062a889de3e14d59f0e7242b4d"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.137259 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.146992 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.149473 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.149646 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.149888 8629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:14.150264 8629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:14.150321 8629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.150373 8629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:14.150400 8629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.156445 8629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42493
I20260924 17:39:14.156507 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42493 every 8 connection(s)
I20260924 17:39:14.156845 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:14.164486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8629
I20260924 17:39:14.164615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:14.164801 392 external_mini_cluster.cc:1428] Setting key 79dbe7f7334b314d769c6c0adfb90752
I20260924 17:39:14.166669 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.157608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54777 (local address 127.0.98.62:45549)
0924 17:39:14.157686 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.157689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.157862 (+ 173us) server_negotiation.cc:407] Connection header received
0924 17:39:14.158824 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.158825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.158835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.158891 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.159578 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.160072 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.161078 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.161278 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.161343 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.163435 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:14.163436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:14.163436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:14.163443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:14.165124 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.165589 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.165590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.165590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.165679 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.166000 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.166001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.166001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.166118 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:14.166194 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:14.166493 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:39:14.166579 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:39:14.166937 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.157566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:54777)
0924 17:39:14.157738 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.157773 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.158718 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.158948 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.158952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.159283 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.159499 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.159505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.160162 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.160162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.160980 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.160982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.161359 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.161360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.161472 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.161945 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.161953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.163353 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:14.165193 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.165200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.165202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.165510 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.165746 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.165747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.165747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.165926 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.166187 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:14.166192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:14.166335 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 17:39:14.166467 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 17:39:14.166627 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260924 17:39:14.167465 8758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:39:14.167582 8758 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:14.167760 8758 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:39:14.168298 8567 ts_manager.cc:194] Registered new tserver with Master: 9569e8a8b8784b8e92817042cdb0de33 (127.0.98.1:42493)
I20260924 17:39:14.168920 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54777
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:39:14.190305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:14.271561 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.271785 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.271811 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.273056 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.273105 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.273133 8763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:14.274698 8763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.274982 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.275261 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:14.280493 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.280519 8763 server_base.cc:1034] running on GCE node
W20260924 17:39:14.280529 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.280470 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.280963 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.281215 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.282382 8763 hybrid_clock.cc:648] HybridClock initialized: now 1790271554282367 us; error 29 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.285013 8763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:39:14.285429 8763 webserver.cc:460] Webserver started at http://127.0.98.2:35617/ using document root <none> and password file <none>
I20260924 17:39:14.285629 8763 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.285667 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.285758 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:14.286770 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "5151894abe934e119179a859011ba253"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "17ab697cd34a9cdf92d5af80170dde39"
server_key_iv: "f08775c4aa26c33364aafc1285180bec"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.287161 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5151894abe934e119179a859011ba253"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "17ab697cd34a9cdf92d5af80170dde39"
server_key_iv: "f08775c4aa26c33364aafc1285180bec"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.288470 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.289189 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.289355 8763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:14.289448 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "5151894abe934e119179a859011ba253"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "17ab697cd34a9cdf92d5af80170dde39"
server_key_iv: "f08775c4aa26c33364aafc1285180bec"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.289501 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.312861 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.315320 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.315450 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.315642 8763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:14.315932 8763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:14.315966 8763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.315989 8763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:14.316001 8763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.321727 8763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33337
I20260924 17:39:14.321779 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33337 every 8 connection(s)
I20260924 17:39:14.322114 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:14.325135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8763
I20260924 17:39:14.325249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:14.325441 392 external_mini_cluster.cc:1428] Setting key 3d814356f960b6f5b8ff85aa3d27f413
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:14.332547 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.322880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33107 (local address 127.0.98.62:45549)
0924 17:39:14.322970 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.322973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.323213 (+ 240us) server_negotiation.cc:407] Connection header received
0924 17:39:14.324218 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.324220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.324232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.324281 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.324939 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.325738 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.326964 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.327237 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.327321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.329698 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:14.329700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:14.329700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:14.329708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:14.331156 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.331579 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.331580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.331580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.331636 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.331882 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.331883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.331883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.332016 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:14.332089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:14.332366 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 17:39:14.332450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:39:14.332770 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.322859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:33107)
0924 17:39:14.323074 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.323119 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.324106 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.324365 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.324369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.324631 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.324819 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.324823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.325836 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.325837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.326872 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.326875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.327330 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.327331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.327501 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.328036 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.328047 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.329590 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:14.331250 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.331256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.331258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.331488 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.331703 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.331704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.331704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.331813 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.332092 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:14.332099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:14.332237 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 17:39:14.332335 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 17:39:14.332527 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260924 17:39:14.333254 8892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:39:14.333364 8892 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:14.333572 8892 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:39:14.333925 8567 ts_manager.cc:194] Registered new tserver with Master: 5151894abe934e119179a859011ba253 (127.0.98.2:33337)
I20260924 17:39:14.334254 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:39:14.351118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:14.431910 8897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.432139 8897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.432164 8897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.433372 8897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.433420 8897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.433449 8897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:14.435015 8897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.435292 8897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.435530 8897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:14.440948 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.440972 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.441110 8897 server_base.cc:1034] running on GCE node
W20260924 17:39:14.440972 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.441414 8897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.441717 8897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.442884 8897 hybrid_clock.cc:648] HybridClock initialized: now 1790271554442865 us; error 30 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.445418 8897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:39:14.445854 8897 webserver.cc:460] Webserver started at http://127.0.98.3:33645/ using document root <none> and password file <none>
I20260924 17:39:14.446100 8897 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.446167 8897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.446311 8897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:14.447449 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "f425276ae41849a3a007aa92c1b71b99"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "bf3d8f1228483279503d2394e7bae971"
server_key_iv: "f0467eaa8f598e515553fc0e99d17978"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.447803 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f425276ae41849a3a007aa92c1b71b99"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "bf3d8f1228483279503d2394e7bae971"
server_key_iv: "f0467eaa8f598e515553fc0e99d17978"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.449101 8897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.449783 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.449946 8897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:14.450044 8897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "f425276ae41849a3a007aa92c1b71b99"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "bf3d8f1228483279503d2394e7bae971"
server_key_iv: "f0467eaa8f598e515553fc0e99d17978"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.450124 8897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.463804 8897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.465867 8897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.466034 8897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.466277 8897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:14.466655 8897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:14.466722 8897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.466766 8897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:14.466794 8897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.472816 8897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37251
I20260924 17:39:14.472858 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37251 every 8 connection(s)
I20260924 17:39:14.473184 8897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:14.475966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 8897
I20260924 17:39:14.476111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:14.476372 392 external_mini_cluster.cc:1428] Setting key 9517a538026218537a1709becd90c35b
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:14.483285 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.473891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58593 (local address 127.0.98.62:45549)
0924 17:39:14.473965 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.473967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.474169 (+ 202us) server_negotiation.cc:407] Connection header received
0924 17:39:14.475225 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.475226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.475239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.475285 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.476119 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.476812 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.478035 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.478229 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.478289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.480519 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:14.480521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:14.480521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:14.480528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:14.481927 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.482343 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.482344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.482345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.482406 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:14.482632 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:14.482632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:14.482633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:14.482743 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:14.482795 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:14.483145 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 17:39:14.483219 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:39:14.483444 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.473882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:58593)
0924 17:39:14.474055 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.474092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.475102 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.475357 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.475361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.475646 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.475933 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.475941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.476933 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.476934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.477953 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.477956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.478303 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.478303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.478406 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.478988 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.478997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.480434 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:14.482017 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.482023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.482025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.482259 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.482475 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:14.482476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:14.482476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:14.482577 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:14.482796 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:14.482800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:14.482924 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 17:39:14.483010 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 17:39:14.483212 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260924 17:39:14.483974 9026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:39:14.484079 9026 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:14.484265 9026 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:39:14.484678 8567 ts_manager.cc:194] Registered new tserver with Master: f425276ae41849a3a007aa92c1b71b99 (127.0.98.3:37251)
I20260924 17:39:14.485190 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58593
I20260924 17:39:14.488366 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 17:39:14.562088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8629
I20260924 17:39:14.566978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8763
I20260924 17:39:14.571661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8897
I20260924 17:39:14.576293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 8535
I20260924 17:39:14.581626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=46031
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:14.664108 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.664336 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.664359 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.665596 9035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:14.665647 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.665661 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.665674 9035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:14.665686 9035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:14.667271 9035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.667539 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.667774 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:14.673130 9035 server_base.cc:1034] running on GCE node
W20260924 17:39:14.673086 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.673095 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.673061 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.673556 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.673908 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.675138 9035 hybrid_clock.cc:648] HybridClock initialized: now 1790271554675130 us; error 39 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.677814 9035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:39:14.678272 9035 webserver.cc:460] Webserver started at http://127.0.98.62:46031/ using document root <none> and password file <none>
I20260924 17:39:14.678506 9035 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.678588 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.680044 9035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.680728 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.680884 9035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:14.680982 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "52c0b3050a1c4d918ac0d35fa721a547"
format_stamp: "Formatted at 2026-09-24 17:39:13 on dist-test-slave-d0pf"
server_key: "70111f5c915e0729ce1df0836e34e1ef"
server_key_iv: "752b42837cfcab9b68018c45c0edf97e"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.681340 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.697382 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.699589 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.699761 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.703788 9035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260924 17:39:14.703871 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260924 17:39:14.704424 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:14.705704 9103 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:39:14.706358 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Bootstrap starting.
I20260924 17:39:14.706768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9035
I20260924 17:39:14.707111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42493
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=34531
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:39:14.715554 9103 log.cc:824] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:14.718266 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:39:14.718648 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Bootstrap complete.
I20260924 17:39:14.721732 9103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:14.722062 9103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b3050a1c4d918ac0d35fa721a547, State: Initialized, Role: FOLLOWER
I20260924 17:39:14.722242 9103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:14.722321 9103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:14.722368 9103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:14.722419 9103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:39:14.723738 9103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:14.723861 9103 leader_election.cc:302] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c0b3050a1c4d918ac0d35fa721a547; no voters:
I20260924 17:39:14.724056 9103 leader_election.cc:288] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:39:14.724172 9107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:39:14.724370 9107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [term 2 LEADER]: Becoming Leader. State: Replica: 52c0b3050a1c4d918ac0d35fa721a547, State: Running, Role: LEADER
I20260924 17:39:14.724501 9103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:14.724532 9107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:39:14.724959 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c0b3050a1c4d918ac0d35fa721a547. Latest consensus state: current_term: 2 leader_uuid: "52c0b3050a1c4d918ac0d35fa721a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:39:14.725054 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:14.724948 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52c0b3050a1c4d918ac0d35fa721a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0b3050a1c4d918ac0d35fa721a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:39:14.725232 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:14.725564 9115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:14.726095 9115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:14.726223 9115 catalog_manager.cc:1259] Loaded cluster ID: 1d57fb9f17c34d32a7f8d07621a7f8d5
I20260924 17:39:14.726262 9115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:14.727450 9115 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:14.727624 9115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52c0b3050a1c4d918ac0d35fa721a547: Loaded TSK: 0
I20260924 17:39:14.727839 9115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:39:14.795300 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.795548 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.795588 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.796840 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.796908 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.796962 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:14.798497 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34531
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.798772 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.799058 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:14.805106 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.805253 9105 server_base.cc:1034] running on GCE node
W20260924 17:39:14.805115 9132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.805128 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.805605 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.805884 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.807104 9105 hybrid_clock.cc:648] HybridClock initialized: now 1790271554807081 us; error 31 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.809859 9105 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 17:39:14.810438 9105 webserver.cc:460] Webserver started at http://127.0.98.1:34531/ using document root <none> and password file <none>
I20260924 17:39:14.810714 9105 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.810796 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.812314 9105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.812968 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.813170 9105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.813268 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "9569e8a8b8784b8e92817042cdb0de33"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "53f1cddd19611b675cb64620f5932d78"
server_key_iv: "232e4a062a889de3e14d59f0e7242b4d"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.813632 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.823081 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.825549 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.825736 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.826077 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:14.826448 9105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:14.826505 9105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.826557 9105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:14.826583 9105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.833892 9105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42493
I20260924 17:39:14.833982 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42493 every 8 connection(s)
I20260924 17:39:14.834359 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.842953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9105
I20260924 17:39:14.843304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33337
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=35617
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.844488 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.835172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:39493)
0924 17:39:14.835365 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.835417 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.836046 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.836711 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.836715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.837311 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.837532 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.837537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.839311 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.839312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.840139 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.840141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.840937 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.840938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.841049 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.841638 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.841648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.844233 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260924 17:39:14.844755 9253 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45549: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 17:39:14.844781 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.835200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39493 (local address 127.0.98.62:45549)
0924 17:39:14.836258 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.836267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.836304 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:39:14.836440 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.836446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.836500 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.836642 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.837679 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.839204 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.840238 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.840847 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.840955 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.844373 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39493: BlockingRecv error: recv got EOF from 127.0.98.1:39493 (error 108)
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":80,"threads_started":1}
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.849757 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.844980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:43327)
0924 17:39:14.845201 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.845213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.845274 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.845406 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.845406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.845549 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.845597 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.845598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.846249 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.846251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.846787 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.846789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.847272 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.847273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.847342 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.847793 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.847800 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.849662 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
W20260924 17:39:14.849812 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.845145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43327 (local address 127.0.98.62:45549)
0924 17:39:14.845199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.845201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.845271 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:39:14.845339 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.845340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.845352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.845404 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.845690 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.846166 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.846894 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.847185 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.847273 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.849745 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43327: BlockingRecv error: recv got EOF from 127.0.98.1:43327 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.854894 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.850348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:52601)
0924 17:39:14.850431 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.850443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.850500 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.850657 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.850658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.850788 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.850822 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.850823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.851550 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.851551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.852218 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.852221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.852596 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.852597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.852650 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.853113 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.853119 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.854798 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 17:39:14.854945 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.850395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52601 (local address 127.0.98.62:45549)
0924 17:39:14.850434 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.850437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.850484 (+ 47us) server_negotiation.cc:407] Connection header received
0924 17:39:14.850562 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.850563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.850573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.850639 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.850896 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.851452 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.852308 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.852510 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.852596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.854879 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52601: BlockingRecv error: recv got EOF from 127.0.98.1:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 17:39:14.855331 9253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:39:14.930716 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:14.930922 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:14.930948 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:14.932161 9257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:14.932211 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:14.932240 9257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:14.933629 9257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35617
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:14.933837 9257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:14.934021 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:14.939422 9257 server_base.cc:1034] running on GCE node
W20260924 17:39:14.939393 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.939404 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:14.939393 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:14.939826 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:14.940052 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:14.941219 9257 hybrid_clock.cc:648] HybridClock initialized: now 1790271554941205 us; error 31 us; skew 500 ppm
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:14.944223 9257 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 17:39:14.944752 9257 webserver.cc:460] Webserver started at http://127.0.98.2:35617/ using document root <none> and password file <none>
I20260924 17:39:14.944988 9257 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:14.945046 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:14.946441 9257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:14.947016 9273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.947233 9257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:39:14.947330 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "5151894abe934e119179a859011ba253"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "17ab697cd34a9cdf92d5af80170dde39"
server_key_iv: "f08775c4aa26c33364aafc1285180bec"
server_key_version: "encryptionkey@0"
I20260924 17:39:14.947691 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:14.964270 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:14.966478 9257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:14.966634 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:14.966907 9257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:14.967293 9257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:14.967334 9257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.967361 9257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:14.967378 9257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:14.974123 9257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33337
I20260924 17:39:14.974165 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33337 every 8 connection(s)
I20260924 17:39:14.974534 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:14.979240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9257
I20260924 17:39:14.979557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37251
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=33645
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:39:14.982616 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.975292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54149 (local address 127.0.98.62:45549)
0924 17:39:14.975360 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.975362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.975579 (+ 217us) server_negotiation.cc:407] Connection header received
0924 17:39:14.976130 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.976131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.976148 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.976206 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.977195 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.977692 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.978704 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.978930 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.979010 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.982568 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54149: BlockingRecv error: recv got EOF from 127.0.98.2:54149 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:39:14.982709 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.975287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:54149)
0924 17:39:14.975454 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.975494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.976011 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.976280 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.976284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.976866 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.977082 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.977087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.977790 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.977791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.978614 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.978616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.979017 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.979018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.979177 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.979834 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.979846 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.982459 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260924 17:39:14.982975 9387 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45549: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.987846 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.983306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:50077)
0924 17:39:14.983352 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.983362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.983420 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.983562 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.983563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.983704 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.983748 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.983750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.984409 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.984410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.984980 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.984982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.985337 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.985338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.985405 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.985855 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.985860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.987759 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:39:14.987890 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.983372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50077 (local address 127.0.98.62:45549)
0924 17:39:14.983438 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.983440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.983449 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:14.983474 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.983475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.983484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.983557 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.983831 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.984328 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.985062 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.985253 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.985337 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.987827 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50077: BlockingRecv error: recv got EOF from 127.0.98.2:50077 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:14 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:14.992535 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:14.988172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:32839)
0924 17:39:14.988213 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:14.988230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:14.988262 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:14.988504 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:14.988505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:14.988623 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:14.988651 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.988652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.989275 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.989276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.989759 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:14.989761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.990131 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.990132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:14.990191 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.990632 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:14.990637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:14.992452 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 17:39:14.992616 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:14.988251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32839 (local address 127.0.98.62:45549)
0924 17:39:14.988385 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:14.988386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:14.988393 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:39:14.988410 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:14.988411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:14.988418 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:14.988485 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:14.988725 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.989187 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.989840 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:14.990045 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:14.990129 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:14.992542 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32839: BlockingRecv error: recv got EOF from 127.0.98.2:32839 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260924 17:39:14.992797 9387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:39:15.065883 9390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:15.066099 9390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:15.066125 9390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:15.067339 9390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:15.067384 9390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:15.067411 9390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:15.068830 9390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33645
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:15.069023 9390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:15.069211 9390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:15.074420 9395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:15.074455 9390 server_base.cc:1034] running on GCE node
W20260924 17:39:15.074502 9398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:15.074406 9396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:15.074838 9390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:15.075158 9390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:15.076336 9390 hybrid_clock.cc:648] HybridClock initialized: now 1790271555076327 us; error 35 us; skew 500 ppm
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:15.079015 9390 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 17:39:15.079506 9390 webserver.cc:460] Webserver started at http://127.0.98.3:33645/ using document root <none> and password file <none>
I20260924 17:39:15.079715 9390 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:15.079769 9390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:15.081125 9390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:15.081722 9405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:15.082028 9390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:39:15.082118 9390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "f425276ae41849a3a007aa92c1b71b99"
format_stamp: "Formatted at 2026-09-24 17:39:14 on dist-test-slave-d0pf"
server_key: "bf3d8f1228483279503d2394e7bae971"
server_key_iv: "f0467eaa8f598e515553fc0e99d17978"
server_key_version: "encryptionkey@0"
I20260924 17:39:15.082465 9390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:15.099561 9390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:15.101971 9390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:15.102094 9390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:15.102370 9390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:15.102701 9390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:15.102769 9390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:15.102811 9390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:15.102837 9390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:15.109361 9390 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37251
I20260924 17:39:15.109434 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37251 every 8 connection(s)
I20260924 17:39:15.109793 9390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:15.115209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9390
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:39:15.117975 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:15.110565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46109 (local address 127.0.98.62:45549)
0924 17:39:15.110656 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.110658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.110866 (+ 208us) server_negotiation.cc:407] Connection header received
0924 17:39:15.111430 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.111431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.111441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.111508 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.112446 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.112946 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.113978 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.114177 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.114237 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.117930 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46109: BlockingRecv error: recv got EOF from 127.0.98.3:46109 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 17:39:15.118079 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.110562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:46109)
0924 17:39:15.110736 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:15.110780 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:15.111341 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:15.111563 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:15.111567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:15.112141 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:15.112341 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.112346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.113042 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.113044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.113885 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.113887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.114272 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.114273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.114393 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.115008 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:15.115016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:15.117858 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
W20260924 17:39:15.118316 9519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45549: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:15.123334 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.118558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:48385)
0924 17:39:15.118610 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:15.118621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:15.118754 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:15.118879 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:15.118880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:15.119079 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:15.119148 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.119151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.120008 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.120009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.120510 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.120512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.120899 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.120900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.120976 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.121325 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:15.121331 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:15.123265 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 17:39:15.123411 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:15.118641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48385 (local address 127.0.98.62:45549)
0924 17:39:15.118680 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.118683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.118693 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:15.118807 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.118808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.118817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.118872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.119250 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.119919 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.120593 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.120823 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.120876 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.123341 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48385: BlockingRecv error: recv got EOF from 127.0.98.3:48385 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:15.124337 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.115754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.98.62:45549)
0924 17:39:15.115895 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.115897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.116040 (+ 143us) server_negotiation.cc:407] Connection header received
0924 17:39:15.116067 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.116068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.116078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.116136 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.117174 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.117680 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.119006 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.119324 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.119382 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.120694 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:15.120698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:15.120698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:15.120710 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:15.122472 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:15.122815 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:15.122819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:15.122819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:15.122950 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:15.123215 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:15.123216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:15.123216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:15.123841 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:15.123992 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:15.124104 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 17:39:15.124219 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:15.135586 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.123600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:52535)
0924 17:39:15.123642 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:15.123664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:15.123694 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:15.123826 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:15.123827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:15.123951 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:15.123977 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.123978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.124980 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.124980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.125671 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.125673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.126021 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.126022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.126073 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.133514 (+ 7441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:15.133524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:15.135488 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 17:39:15.135772 9519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:39:15.135807 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:15.123644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52535 (local address 127.0.98.62:45549)
0924 17:39:15.123678 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.123679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.123688 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:15.123750 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.123751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.123758 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.123821 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.124062 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.124913 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.125749 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.125949 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.126024 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.135618 (+ 9594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52535: BlockingRecv error: recv got EOF from 127.0.98.3:52535 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:15.860883 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.855810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:37925)
0924 17:39:15.856004 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:15.856018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:15.856085 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:15.856253 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:15.856254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:15.856412 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:15.856481 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.856483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.857209 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.857210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.857765 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.857767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.858125 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.858126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.858175 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.858662 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:15.858667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:15.860814 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
W20260924 17:39:15.861074 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:15.855925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37925 (local address 127.0.98.62:45549)
0924 17:39:15.856057 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.856060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.856073 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:39:15.856136 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.856137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.856147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.856209 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.856579 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.857098 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.857857 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.858052 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.858116 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.860987 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37925: BlockingRecv error: recv got EOF from 127.0.98.1:37925 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:15 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:15.998394 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:15.993342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:54533)
0924 17:39:15.993506 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:15.993520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:15.993612 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:15.993781 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:15.993782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:15.993967 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:15.994013 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.994015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.994783 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.994785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.995415 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:15.995417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.995783 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.995784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:15.995849 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.996324 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:15.996329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:15.998319 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
W20260924 17:39:15.998584 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:15.993473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54533 (local address 127.0.98.62:45549)
0924 17:39:15.993528 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:15.993530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:15.993607 (+ 77us) server_negotiation.cc:407] Connection header received
0924 17:39:15.993681 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:15.993682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:15.993698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:15.993773 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:15.994132 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.994694 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.995512 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:15.995709 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:15.995786 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:15.998497 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54533: BlockingRecv error: recv got EOF from 127.0.98.2:54533 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:16.141179 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:16.136246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:51953)
0924 17:39:16.136397 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:16.136411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:16.136479 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:16.136658 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:16.136659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:16.136911 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:16.136975 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:16.136977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.137692 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.137693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:16.138247 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:16.138248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.138625 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.138625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:16.138682 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:16.139252 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:16.139257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:16.141070 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260924 17:39:16.141222 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:16.136372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51953 (local address 127.0.98.62:45549)
0924 17:39:16.136448 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:16.136451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:16.136462 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:16.136577 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:16.136578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:16.136591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:16.136654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:16.137087 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.137607 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.138344 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.138542 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.138623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:16.141156 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51953: BlockingRecv error: recv got EOF from 127.0.98.3:51953 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:16.865942 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:16.861402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:57087)
0924 17:39:16.861543 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:16.861554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:16.861602 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:16.861765 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:16.861766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:16.861924 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:16.861977 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:16.861978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.862665 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.862665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:16.863275 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:16.863278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.863648 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.863648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:16.863687 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:16.864150 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:16.864154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:16.865882 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
W20260924 17:39:16.866111 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:16.861473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57087 (local address 127.0.98.62:45549)
0924 17:39:16.861610 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:16.861612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:16.861623 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:16.861649 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:16.861650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:16.861668 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:16.861738 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:16.862051 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.862590 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.863375 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.863569 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:16.863635 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:16.865997 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57087: BlockingRecv error: recv got EOF from 127.0.98.1:57087 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:16 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:17.003417 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:16.998871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:50783)
0924 17:39:16.998984 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:16.998996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:16.999084 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:16.999210 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:16.999210 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:16.999374 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:16.999421 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:16.999423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.000036 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.000037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.000550 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:17.000551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.000862 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.000862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.000901 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.001316 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:17.001321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:17.003348 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
W20260924 17:39:17.003551 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:16.998926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50783 (local address 127.0.98.62:45549)
0924 17:39:16.998990 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:16.998992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:16.999059 (+ 67us) server_negotiation.cc:407] Connection header received
0924 17:39:16.999139 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:16.999140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:16.999150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:16.999199 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:16.999484 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:16.999969 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.000624 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.000800 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.000853 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.003469 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50783: BlockingRecv error: recv got EOF from 127.0.98.2:50783 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 17:39:17 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:17 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:17.146495 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:17.141735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:60119)
0924 17:39:17.141891 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:17.141919 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:17.141980 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:17.142130 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:17.142130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:17.142329 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:17.142375 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:17.142378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.143091 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.143092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.143637 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:17.143639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.144000 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.144001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.144060 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.144567 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:17.144572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:17.146401 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260924 17:39:17.146555 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:17.141872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60119 (local address 127.0.98.62:45549)
0924 17:39:17.141945 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:17.141947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:17.141960 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:39:17.142035 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:17.142036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:17.142050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:17.142124 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:17.142462 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.142972 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.143727 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.143924 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.144003 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.146492 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60119: BlockingRecv error: recv got EOF from 127.0.98.3:60119 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 17:39:17 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:17 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:17.871543 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:17.866538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:38421)
0924 17:39:17.866685 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:17.866697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:17.866757 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:17.866900 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:17.866900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:17.867086 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:17.867135 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:17.867137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.867860 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.867862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.868398 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:17.868400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.868793 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.868794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:17.868837 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.869509 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:17.869514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:17.871482 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260924 17:39:17.871733 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:17.866615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38421 (local address 127.0.98.62:45549)
0924 17:39:17.866766 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:17.866769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:17.866780 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:17.866804 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:17.866805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:17.866815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:17.866884 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:17.867228 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.867746 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.868515 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:17.868719 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:17.868783 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:17.871656 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38421: BlockingRecv error: recv got EOF from 127.0.98.1:38421 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:18.008679 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:18.003937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:54233)
0924 17:39:18.004077 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:18.004088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:18.004177 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:18.004338 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:18.004339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:18.004491 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:18.004539 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.004540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.005189 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.005191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.005715 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.005717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.006080 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.006081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.006130 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.006691 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:18.006696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:18.008618 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260924 17:39:18.008809 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:18.003977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54233 (local address 127.0.98.62:45549)
0924 17:39:18.004031 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:18.004033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:18.004160 (+ 127us) server_negotiation.cc:407] Connection header received
0924 17:39:18.004253 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:18.004254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:18.004267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:18.004328 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:18.004603 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.005089 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.005799 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.005988 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.006042 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.008727 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54233: BlockingRecv error: recv got EOF from 127.0.98.2:54233 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:18.151470 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:18.146975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:58573)
0924 17:39:18.147169 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:18.147179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:18.147239 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:18.147364 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:18.147364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:18.147525 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:18.147575 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.147576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.148169 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.148170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.148682 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.148684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.149055 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.149056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.149095 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.149513 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:18.149517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:18.151401 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
W20260924 17:39:18.151576 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:18.147026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58573 (local address 127.0.98.62:45549)
0924 17:39:18.147124 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:18.147126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:18.147242 (+ 116us) server_negotiation.cc:407] Connection header received
0924 17:39:18.147287 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:18.147288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:18.147301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:18.147348 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:18.147634 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.148106 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.148767 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.148973 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.149024 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.151484 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58573: BlockingRecv error: recv got EOF from 127.0.98.3:58573 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:18 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:18.877161 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:18.872098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:58127)
0924 17:39:18.872287 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:18.872298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:18.872354 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:18.872495 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:18.872497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:18.872655 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:18.872706 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.872708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.873412 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.873414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.874066 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:18.874070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.874516 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.874517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:18.874561 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.875255 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:18.875260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:18.877085 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
W20260924 17:39:18.877274 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:18.872214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58127 (local address 127.0.98.62:45549)
0924 17:39:18.872339 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:18.872341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:18.872350 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:18.872401 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:18.872401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:18.872412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:18.872476 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:18.872777 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.873282 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.874184 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:18.874417 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:18.874511 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:18.877197 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58127: BlockingRecv error: recv got EOF from 127.0.98.1:58127 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:19.013851 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:19.009182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:48937)
0924 17:39:19.009351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:19.009365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:19.009450 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:19.009583 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:19.009584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:19.009760 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:19.009804 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.009806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.010480 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.010481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.011012 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.011013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.011396 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.011397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.011451 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.011966 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:19.011971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:19.013791 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260924 17:39:19.014003 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:19.009283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48937 (local address 127.0.98.62:45549)
0924 17:39:19.009374 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:19.009376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:19.009475 (+ 99us) server_negotiation.cc:407] Connection header received
0924 17:39:19.009508 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:19.009509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:19.009522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:19.009579 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:19.009888 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.010378 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.011134 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.011327 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.011400 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.013922 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48937: BlockingRecv error: recv got EOF from 127.0.98.2:48937 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:19.156456 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:19.151896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:47455)
0924 17:39:19.152034 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:19.152044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:19.152105 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:19.152233 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:19.152234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:19.152385 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:19.152432 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.152434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.153051 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.153052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.153755 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.153758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.154108 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.154109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.154149 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.154624 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:19.154636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:19.156399 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260924 17:39:19.156589 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:19.151964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47455 (local address 127.0.98.62:45549)
0924 17:39:19.152048 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:19.152050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:19.152107 (+ 57us) server_negotiation.cc:407] Connection header received
0924 17:39:19.152159 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:19.152160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:19.152172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:19.152212 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:19.152494 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.152989 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.153821 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.154050 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.154102 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.156508 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47455: BlockingRecv error: recv got EOF from 127.0.98.3:47455 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:19 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:19.882146 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:19.877663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:56473)
0924 17:39:19.877827 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:19.877838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:19.877891 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:19.878029 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:19.878030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:19.878181 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:19.878232 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.878234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.878854 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.878855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.879392 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:19.879394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.879709 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.879709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:19.879752 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.880180 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:19.880185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:19.882051 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260924 17:39:19.882205 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:19.877752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56473 (local address 127.0.98.62:45549)
0924 17:39:19.877861 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:19.877863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:19.877875 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:39:19.877948 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:19.877949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:19.877959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:19.878015 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:19.878292 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.878790 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.879453 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:19.879648 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:19.879699 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:19.882128 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56473: BlockingRecv error: recv got EOF from 127.0.98.1:56473 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:20.018953 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:20.014435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:36945)
0924 17:39:20.014584 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:20.014595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:20.014645 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:20.014785 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:20.014786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:20.014941 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:20.014987 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.014989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.015643 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.015644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.016265 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.016267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.016618 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.016618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.016654 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.017094 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:20.017115 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:20.018883 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260924 17:39:20.019021 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:20.014597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36945 (local address 127.0.98.62:45549)
0924 17:39:20.014657 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:20.014659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:20.014668 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:20.014703 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:20.014704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:20.014715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:20.014768 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:20.015081 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.015571 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.016329 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.016515 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.016609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.018975 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36945: BlockingRecv error: recv got EOF from 127.0.98.2:36945 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:20.165468 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:20.156906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:58099)
0924 17:39:20.157091 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:20.157104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:20.157187 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:20.157344 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:20.157345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:20.157521 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:20.157567 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.157569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.158227 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.158228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.158773 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.158774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.159198 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.159199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.159271 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.163317 (+ 4046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:20.163322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:20.165399 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
W20260924 17:39:20.165580 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:20.157000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58099 (local address 127.0.98.62:45549)
0924 17:39:20.157070 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:20.157072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:20.157168 (+ 96us) server_negotiation.cc:407] Connection header received
0924 17:39:20.157249 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:20.157250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:20.157266 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:20.157336 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:20.157654 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.158147 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.158858 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.159118 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.159198 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.165486 (+ 6288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58099: BlockingRecv error: recv got EOF from 127.0.98.3:58099 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:20 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:20.887943 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:20.882838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:39725)
0924 17:39:20.882973 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:20.882985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:20.883065 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:20.883218 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:20.883219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:20.883409 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:20.883467 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.883470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.884184 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.884185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.884841 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:20.884844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.885152 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.885153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:20.885203 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.885779 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:20.885784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:20.887828 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260924 17:39:20.888020 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:20.882889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39725 (local address 127.0.98.62:45549)
0924 17:39:20.883027 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:20.883069 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:20.883116 (+ 47us) server_negotiation.cc:407] Connection header received
0924 17:39:20.883138 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:20.883138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:20.883148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:20.883198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:20.883534 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.884092 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.884903 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:20.885090 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:20.885165 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:20.887924 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39725: BlockingRecv error: recv got EOF from 127.0.98.1:39725 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:21.023955 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:21.019384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:48901)
0924 17:39:21.019515 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:21.019526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:21.019590 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:21.019731 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:21.019732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:21.019911 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:21.019959 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.019960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.020580 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.020581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.021107 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.021108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.021423 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.021424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.021464 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.021974 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:21.021980 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:21.023870 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260924 17:39:21.024127 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:21.019451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48901 (local address 127.0.98.62:45549)
0924 17:39:21.019503 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:21.019504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:21.019576 (+ 72us) server_negotiation.cc:407] Connection header received
0924 17:39:21.019646 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:21.019647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:21.019668 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:21.019709 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:21.020024 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.020516 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.021178 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.021360 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.021410 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.024012 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48901: BlockingRecv error: recv got EOF from 127.0.98.2:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:21.170214 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:21.165945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:58527)
0924 17:39:21.166088 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:21.166100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:21.166164 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:21.166294 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:21.166294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:21.166538 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:21.166586 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.166588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.167269 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.167269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.167785 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.167786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.168098 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.168099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.168135 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.168563 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:21.168568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:21.170159 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
W20260924 17:39:21.170315 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:21.166059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58527 (local address 127.0.98.62:45549)
0924 17:39:21.166103 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:21.166106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:21.166151 (+ 45us) server_negotiation.cc:407] Connection header received
0924 17:39:21.166222 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:21.166223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:21.166233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:21.166275 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:21.166641 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.167204 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.167851 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.168038 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.168087 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.170246 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58527: BlockingRecv error: recv got EOF from 127.0.98.3:58527 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:21 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:21.892728 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:21.888492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:42655)
0924 17:39:21.888634 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:21.888644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:21.888694 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:21.888830 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:21.888831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:21.888994 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:21.889042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.889044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.889661 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.889662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.890177 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:21.890179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.890479 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.890480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:21.890519 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.890945 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:21.890949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:21.892660 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
W20260924 17:39:21.892797 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:21.888559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42655 (local address 127.0.98.62:45549)
0924 17:39:21.888686 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:21.888687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:21.888696 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:21.888738 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:21.888739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:21.888748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:21.888807 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:21.889107 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.889596 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.890241 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:21.890419 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:21.890469 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:21.892746 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42655: BlockingRecv error: recv got EOF from 127.0.98.1:42655 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:22.029275 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:22.024500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:59649)
0924 17:39:22.024646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:22.024658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:22.024719 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:22.024910 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:22.024912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:22.025086 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:22.025130 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.025131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.025923 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.025924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.026454 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.026456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.026844 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.026845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.026903 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.027415 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:22.027422 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:22.029215 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
W20260924 17:39:22.029368 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:22.024549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59649 (local address 127.0.98.62:45549)
0924 17:39:22.024593 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:22.024595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:22.024701 (+ 106us) server_negotiation.cc:407] Connection header received
0924 17:39:22.024782 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:22.024783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:22.024796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:22.024872 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:22.025232 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.025831 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.026549 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.026757 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.026845 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.029304 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59649: BlockingRecv error: recv got EOF from 127.0.98.2:59649 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:22.175277 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:22.170684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:43241)
0924 17:39:22.170869 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:22.170879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:22.170990 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:22.171162 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:22.171163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:22.171339 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:22.171386 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.171388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.172071 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.172073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.172594 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.172595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.172960 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.172961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.173024 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.173450 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:22.173454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:22.175219 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
W20260924 17:39:22.175442 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:22.170778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43241 (local address 127.0.98.62:45549)
0924 17:39:22.170849 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:22.170851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:22.170969 (+ 118us) server_negotiation.cc:407] Connection header received
0924 17:39:22.171067 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:22.171068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:22.171083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:22.171156 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:22.171481 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.171982 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.172675 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.172872 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.172946 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.175358 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43241: BlockingRecv error: recv got EOF from 127.0.98.3:43241 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:22 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:22.897619 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:22.893229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:38797)
0924 17:39:22.893359 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:22.893369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:22.893416 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:22.893636 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:22.893637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:22.893791 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:22.893840 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.893842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.894453 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.894454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.894946 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:22.894947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.895285 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.895286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:22.895326 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.895767 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:22.895772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:22.897517 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260924 17:39:22.897666 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:22.893370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38797 (local address 127.0.98.62:45549)
0924 17:39:22.893499 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:22.893501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:22.893516 (+ 15us) server_negotiation.cc:407] Connection header received
0924 17:39:22.893537 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:22.893538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:22.893557 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:22.893617 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:22.893906 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.894394 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.895013 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:22.895225 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:22.895274 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:22.897598 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38797: BlockingRecv error: recv got EOF from 127.0.98.1:38797 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:23.034432 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:23.029790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:55339)
0924 17:39:23.029931 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:23.029941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:23.030006 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:23.030146 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:23.030147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:23.030306 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:23.030351 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.030353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.031138 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.031138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.031678 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.031679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.032081 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.032081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.032124 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.032566 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:23.032570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:23.034367 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260924 17:39:23.034519 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:23.029863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55339 (local address 127.0.98.62:45549)
0924 17:39:23.029929 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:23.029931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:23.030010 (+ 79us) server_negotiation.cc:407] Connection header received
0924 17:39:23.030059 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:23.030060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:23.030072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:23.030126 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:23.030415 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.031075 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.031770 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.031995 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.032062 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.034467 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55339: BlockingRecv error: recv got EOF from 127.0.98.2:55339 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:23.180567 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:23.175799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:43321)
0924 17:39:23.176052 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:23.176071 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:23.176165 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:23.176323 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:23.176324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:23.176511 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:23.176557 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.176559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.177229 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.177230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.177761 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.177763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.178132 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.178133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.178199 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.178656 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:23.178660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:23.180510 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
W20260924 17:39:23.180699 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:23.175908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43321 (local address 127.0.98.62:45549)
0924 17:39:23.176010 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:23.176013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:23.176147 (+ 134us) server_negotiation.cc:407] Connection header received
0924 17:39:23.176226 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:23.176227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:23.176242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:23.176318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:23.176642 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.177135 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.177850 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.178043 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.178130 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.180618 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43321: BlockingRecv error: recv got EOF from 127.0.98.3:43321 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:23 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:23.913756 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:23.898158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:57753)
0924 17:39:23.898273 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:23.898284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:23.898343 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:23.898530 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:23.898532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:23.898756 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:23.898832 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.898834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.899586 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.899587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.900166 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:23.900168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.900535 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.900536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:23.900578 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.911712 (+ 11134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:23.911720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:23.913694 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
W20260924 17:39:23.913923 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:23.898217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57753 (local address 127.0.98.62:45549)
0924 17:39:23.898358 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:23.898361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:23.898372 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:23.898402 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:23.898403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:23.898417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:23.898503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:23.898920 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.899493 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.900265 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:23.900466 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:23.900530 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:23.913817 (+ 13287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57753: BlockingRecv error: recv got EOF from 127.0.98.1:57753 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:24.039407 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:24.035000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:42939)
0924 17:39:24.035167 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:24.035179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:24.035270 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:24.035446 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:24.035447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:24.035620 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:24.035679 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.035681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.036306 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.036306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.036819 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.036820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.037132 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.037132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.037170 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.037595 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:24.037599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:24.039346 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
W20260924 17:39:24.039618 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:24.035094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42939 (local address 127.0.98.62:45549)
0924 17:39:24.035142 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:24.035144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:24.035253 (+ 109us) server_negotiation.cc:407] Connection header received
0924 17:39:24.035356 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:24.035357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:24.035370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:24.035426 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:24.035752 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.036238 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.036888 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.037071 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.037121 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.039538 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42939: BlockingRecv error: recv got EOF from 127.0.98.2:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:24.185757 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:24.181053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:38363)
0924 17:39:24.181186 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:24.181200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:24.181265 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:24.181402 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:24.181403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:24.181558 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:24.181607 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.181609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.182251 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.182252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.182763 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.182764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.183144 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.183144 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.183188 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.183737 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:24.183744 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:24.185679 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260924 17:39:24.185868 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:24.181117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38363 (local address 127.0.98.62:45549)
0924 17:39:24.181201 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:24.181204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:24.181264 (+ 60us) server_negotiation.cc:407] Connection header received
0924 17:39:24.181326 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:24.181327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:24.181339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:24.181382 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:24.181668 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.182184 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.182832 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.183007 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.183068 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.185794 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38363: BlockingRecv error: recv got EOF from 127.0.98.3:38363 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:24 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:24.919209 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:24.914338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:43575)
0924 17:39:24.914484 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:24.914495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:24.914542 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:24.914672 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:24.914685 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:24.914841 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:24.914891 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.914893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.915570 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.915571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.916187 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:24.916190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.916661 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.916662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:24.916711 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.917313 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:24.917322 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:24.919148 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260924 17:39:24.919374 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:24.914393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43575 (local address 127.0.98.62:45549)
0924 17:39:24.914538 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:24.914540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:24.914549 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:24.914587 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:24.914588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:24.914597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:24.914663 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:24.914954 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.915491 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.916363 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:24.916569 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:24.916630 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:24.919264 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43575: BlockingRecv error: recv got EOF from 127.0.98.1:43575 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:25.044708 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:25.040036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:37795)
0924 17:39:25.040195 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:25.040206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:25.040272 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:25.040421 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:25.040421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:25.040580 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:25.040626 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.040627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.041240 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.041241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.041742 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.041743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.042067 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.042068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.042136 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.042764 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:25.042771 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:25.044605 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
W20260924 17:39:25.044760 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:25.040096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37795 (local address 127.0.98.62:45549)
0924 17:39:25.040170 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:25.040172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:25.040276 (+ 104us) server_negotiation.cc:407] Connection header received
0924 17:39:25.040326 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:25.040327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:25.040338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:25.040398 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:25.040689 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.041176 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.041813 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.041995 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.042045 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.044693 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37795: BlockingRecv error: recv got EOF from 127.0.98.2:37795 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:25.195955 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:25.186241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:39281)
0924 17:39:25.186387 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:25.186401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:25.186470 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:25.186594 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:25.186595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:25.186757 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:25.186811 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.186812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.187497 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.187498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.188037 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.188039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.188342 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.188343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.188380 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.193954 (+ 5574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:25.193959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:25.195849 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
W20260924 17:39:25.195992 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:25.186365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39281 (local address 127.0.98.62:45549)
0924 17:39:25.186411 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:25.186413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:25.186473 (+ 60us) server_negotiation.cc:407] Connection header received
0924 17:39:25.186522 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:25.186523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:25.186534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:25.186573 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:25.186875 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.187434 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.188106 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.188282 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.188329 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.195932 (+ 7603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39281: BlockingRecv error: recv got EOF from 127.0.98.3:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:25 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:25.924165 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:25.919654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:38579)
0924 17:39:25.919807 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:25.919819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:25.919881 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:25.920014 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:25.920015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:25.920186 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:25.920236 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.920238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.920939 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.920940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.921480 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:25.921482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.921810 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.921810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:25.921848 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.922294 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:25.922299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:25.924107 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
W20260924 17:39:25.924276 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:25.919715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38579 (local address 127.0.98.62:45549)
0924 17:39:25.919861 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:25.919863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:25.919871 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:25.919929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:25.919930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:25.919939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:25.919996 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:25.920343 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.920859 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.921557 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:25.921746 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:25.921799 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:25.924199 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38579: BlockingRecv error: recv got EOF from 127.0.98.1:38579 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:26.049891 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:26.045242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:35733)
0924 17:39:26.045383 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:26.045395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:26.045470 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:26.045623 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:26.045624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:26.045784 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:26.045832 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.045834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.046470 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.046471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.046990 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.046991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.047343 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.047344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.047384 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.047847 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:26.047852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:26.049831 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260924 17:39:26.050021 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:26.045367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35733 (local address 127.0.98.62:45549)
0924 17:39:26.045430 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:26.045432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:26.045461 (+ 29us) server_negotiation.cc:407] Connection header received
0924 17:39:26.045533 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:26.045534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:26.045545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:26.045605 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:26.045896 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.046389 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.047089 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.047278 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.047330 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.049940 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35733: BlockingRecv error: recv got EOF from 127.0.98.2:35733 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:26.201771 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:26.196486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:50845)
0924 17:39:26.196714 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:26.196733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:26.196880 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:26.197070 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:26.197071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:26.197449 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:26.197531 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.197536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.198301 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.198303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.198844 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.198845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.199291 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.199292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.199343 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.199861 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:26.199865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:26.201670 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
W20260924 17:39:26.201826 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:26.196761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50845 (local address 127.0.98.62:45549)
0924 17:39:26.196849 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:26.196852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:26.196866 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:39:26.196958 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:26.196960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:26.196979 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:26.197062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:26.197682 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.198202 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.198946 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.199216 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.199293 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.201761 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50845: BlockingRecv error: recv got EOF from 127.0.98.3:50845 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:26 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:26.930081 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:26.924689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:40751)
0924 17:39:26.924935 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:26.924948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:26.925010 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:26.925214 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:26.925215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:26.925481 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:26.925532 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.925535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.926489 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.926490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.927080 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:26.927082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.927456 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.927457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:26.927511 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.928119 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:26.928124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:26.930022 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":200,"threads_started":1}
W20260924 17:39:26.930244 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:26.924875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40751 (local address 127.0.98.62:45549)
0924 17:39:26.925047 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:26.925049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:26.925065 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:39:26.925108 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:26.925109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:26.925123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:26.925204 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:26.925652 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.926415 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.927149 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:26.927348 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:26.927415 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:26.930136 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40751: BlockingRecv error: recv got EOF from 127.0.98.1:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:27.055493 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:27.050456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:41915)
0924 17:39:27.050626 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:27.050639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:27.050741 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:27.050891 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:27.050891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:27.051123 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:27.051168 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.051170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.051911 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.051913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.052497 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.052500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.052895 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.052896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.052955 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.053445 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:27.053450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:27.055419 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
W20260924 17:39:27.055605 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:27.050554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41915 (local address 127.0.98.62:45549)
0924 17:39:27.050630 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:27.050633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:27.050723 (+ 90us) server_negotiation.cc:407] Connection header received
0924 17:39:27.050810 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:27.050811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:27.050823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:27.050873 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:27.051265 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.051798 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.052590 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.052807 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.052895 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.055533 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41915: BlockingRecv error: recv got EOF from 127.0.98.2:41915 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:27.207167 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:27.202270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:46085)
0924 17:39:27.202478 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:27.202488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:27.202546 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:27.202664 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:27.202665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:27.202820 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:27.202868 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.202869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.203813 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.203814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.204335 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.204337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.204733 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.204735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.204780 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.205229 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:27.205234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:27.207100 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
W20260924 17:39:27.207240 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:27.202367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46085 (local address 127.0.98.62:45549)
0924 17:39:27.202432 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:27.202434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:27.202549 (+ 115us) server_negotiation.cc:407] Connection header received
0924 17:39:27.202588 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:27.202588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:27.202600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:27.202649 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:27.202925 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.203727 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.204428 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.204640 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.204706 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.207190 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46085: BlockingRecv error: recv got EOF from 127.0.98.3:46085 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:27 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:27.935325 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:27.930605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:33245)
0924 17:39:27.930734 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:27.930751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:27.930805 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:27.931000 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:27.931001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:27.931181 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:27.931228 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.931229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.931869 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.931870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.932380 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:27.932382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.932985 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.932986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:27.933040 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.933522 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:27.933526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:27.935268 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260924 17:39:27.935547 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:27.930720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33245 (local address 127.0.98.62:45549)
0924 17:39:27.930862 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:27.930864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:27.930873 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:39:27.930906 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:27.930907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:27.930917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:27.930984 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:27.931295 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.931801 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.932477 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:27.932884 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:27.932958 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:27.935420 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33245: BlockingRecv error: recv got EOF from 127.0.98.1:33245 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:28.060344 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:28.056041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:39917)
0924 17:39:28.056173 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:28.056198 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:28.056246 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:28.056399 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:28.056400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:28.056559 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:28.056607 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.056609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.057230 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.057231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.057759 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.057761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.058093 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.058094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.058131 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.058556 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:28.058560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:28.060282 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260924 17:39:28.060510 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:28.056170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39917 (local address 127.0.98.62:45549)
0924 17:39:28.056236 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:28.056243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:28.056265 (+ 22us) server_negotiation.cc:407] Connection header received
0924 17:39:28.056295 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:28.056296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:28.056308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:28.056381 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:28.056671 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.057152 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.057846 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.058028 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.058088 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.060410 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39917: BlockingRecv error: recv got EOF from 127.0.98.2:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:28.213079 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:28.207932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:33685)
0924 17:39:28.208079 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:28.208093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:28.208160 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:28.208406 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:28.208407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:28.208625 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:28.208683 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.208688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.209336 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.209337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.209882 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.209884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.210493 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.210495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.210565 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.211173 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:28.211178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:28.213019 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260924 17:39:28.213248 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:28.208019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33685 (local address 127.0.98.62:45549)
0924 17:39:28.208084 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:28.208087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:28.208160 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:39:28.208278 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:28.208279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:28.208295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:28.208357 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:28.208758 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.209250 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.209968 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.210168 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.210225 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.213139 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33685: BlockingRecv error: recv got EOF from 127.0.98.3:33685 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:28 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:28.940289 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:28.935847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:54277)
0924 17:39:28.935962 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:28.935973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:28.936030 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:28.936156 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:28.936157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:28.936317 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:28.936366 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.936367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.937014 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.937015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.937559 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:28.937561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.937899 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.937900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:28.937944 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.938386 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:28.938391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:28.940232 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
W20260924 17:39:28.940418 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:28.935884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54277 (local address 127.0.98.62:45549)
0924 17:39:28.935986 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:28.935988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:28.936022 (+ 34us) server_negotiation.cc:407] Connection header received
0924 17:39:28.936087 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:28.936088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:28.936098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:28.936153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:28.936428 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.936921 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.937642 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:28.937829 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:28.937886 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:28.940328 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54277: BlockingRecv error: recv got EOF from 127.0.98.1:54277 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":41,"threads_started":1}
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:29.065590 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:29.060856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:52635)
0924 17:39:29.060993 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:29.061003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:29.061086 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:29.061261 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:29.061262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:29.061442 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:29.061487 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.061489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.062157 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.062158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.062696 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.062697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.063087 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.063088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.063154 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.063618 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:29.063623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:29.065478 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260924 17:39:29.065636 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:29.060992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52635 (local address 127.0.98.62:45549)
0924 17:39:29.061065 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:29.061067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:29.061078 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:29.061164 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:29.061166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:29.061183 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:29.061256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:29.061568 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.062067 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.062784 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.062981 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.063082 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.065570 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52635: BlockingRecv error: recv got EOF from 127.0.98.2:52635 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:29.218236 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:29.213621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:43517)
0924 17:39:29.213831 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:29.213842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:29.213897 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:29.214029 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:29.214030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:29.214189 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:29.214248 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.214250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.214898 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.214899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.215523 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.215525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.215889 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.215890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.215935 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.216419 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:29.216425 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:29.218166 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":123,"threads_started":1}
W20260924 17:39:29.218407 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:29.213723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43517 (local address 127.0.98.62:45549)
0924 17:39:29.213770 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:29.213772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:29.213884 (+ 112us) server_negotiation.cc:407] Connection header received
0924 17:39:29.213954 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:29.213955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:29.213967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:29.214008 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:29.214316 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.214808 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.215610 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.215823 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.215877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.218308 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43517: BlockingRecv error: recv got EOF from 127.0.98.3:43517 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:29 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:29.945358 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:29.940819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:54753)
0924 17:39:29.940990 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:29.941002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:29.941078 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:29.941208 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:29.941208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:29.941368 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:29.941415 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.941417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.942048 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.942049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.942571 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:29.942572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.942921 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.942921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:29.942959 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.943471 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:29.943476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:29.945295 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
W20260924 17:39:29.945500 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:29.940904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54753 (local address 127.0.98.62:45549)
0924 17:39:29.941028 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:29.941030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:29.941068 (+ 38us) server_negotiation.cc:407] Connection header received
0924 17:39:29.941121 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:29.941122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:29.941132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:29.941188 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:29.941481 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.941970 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.942653 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:29.942849 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:29.942915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:29.945410 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54753: BlockingRecv error: recv got EOF from 127.0.98.1:54753 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
Sep 24 17:39:30 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:39:30 dist-test-slave-d0pf krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:39:30.071199 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.066181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:43443)
0924 17:39:30.066383 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:30.066394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:30.066483 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:30.066647 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:30.066648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:30.066833 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:30.066880 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.066881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.067598 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.067599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.068130 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.068132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.068528 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.068529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.068583 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.069097 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:30.069102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:30.071127 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45549: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
W20260924 17:39:30.071296 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:30.066279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43443 (local address 127.0.98.62:45549)
0924 17:39:30.066354 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.066356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.066463 (+ 107us) server_negotiation.cc:407] Connection header received
0924 17:39:30.066552 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.066553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.066568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.066639 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.066966 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.067516 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.068229 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.068427 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.068530 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.071222 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43443: BlockingRecv error: recv got EOF from 127.0.98.2:43443 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:39:30.124606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9105
I20260924 17:39:30.129154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9257
I20260924 17:39:30.133461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9390
I20260924 17:39:30.137677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9035
2026-09-24T17:39:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17474 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T17:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:30Z Disabled control of system clock
I20260924 17:39:30.166321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33449
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 17:39:30.251823 9586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.252063 9586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.252085 9586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.253245 9586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:30.253284 9586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.253296 9586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.253309 9586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:30.253321 9586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:30.254632 9586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.254848 9586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.255141 9586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:30.260546 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.260574 9586 server_base.cc:1034] running on GCE node
W20260924 17:39:30.260528 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.260524 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.261059 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:30.261405 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:30.262593 9586 hybrid_clock.cc:648] HybridClock initialized: now 1790271570262566 us; error 40 us; skew 500 ppm
I20260924 17:39:30.268356 9586 webserver.cc:460] Webserver started at http://127.0.98.62:44591/ using document root <none> and password file <none>
I20260924 17:39:30.268656 9586 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:30.268733 9586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:30.268888 9586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:30.270026 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "c04032e9504f499ca52c702d5dd5ccf4"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "f83ca88083b4f26a4cca0ec281395d76"
server_key_iv: "5677bb5cd7a58d600cd35416927ec1dd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.270416 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "c04032e9504f499ca52c702d5dd5ccf4"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "f83ca88083b4f26a4cca0ec281395d76"
server_key_iv: "5677bb5cd7a58d600cd35416927ec1dd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.271880 9586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:39:30.272737 9600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.272923 9586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:30.273047 9586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "c04032e9504f499ca52c702d5dd5ccf4"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "f83ca88083b4f26a4cca0ec281395d76"
server_key_iv: "5677bb5cd7a58d600cd35416927ec1dd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.273129 9586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:30.292001 9586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:30.292342 9586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:30.292519 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:30.296797 9586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33449
I20260924 17:39:30.296835 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33449 every 8 connection(s)
I20260924 17:39:30.297253 9586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:30.297955 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:30.300868 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4: Bootstrap starting.
I20260924 17:39:30.301472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9586
I20260924 17:39:30.301582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:30.301596 9653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:30.301823 392 external_mini_cluster.cc:1428] Setting key d21682aaa99ed84066e024e8ab13775c
I20260924 17:39:30.301946 9653 log.cc:824] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:30.302798 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4: No bootstrap required, opened a new log
I20260924 17:39:30.305236 9653 raft_consensus.cc:348] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } }
I20260924 17:39:30.305433 9653 raft_consensus.cc:374] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:30.305466 9653 raft_consensus.cc:729] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04032e9504f499ca52c702d5dd5ccf4, State: Initialized, Role: FOLLOWER
I20260924 17:39:30.305586 9653 consensus_queue.cc:260] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } }
I20260924 17:39:30.305673 9653 raft_consensus.cc:388] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:30.305704 9653 raft_consensus.cc:482] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:30.305750 9653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:30.306383 9653 raft_consensus.cc:504] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } }
I20260924 17:39:30.306502 9653 leader_election.cc:302] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c04032e9504f499ca52c702d5dd5ccf4; no voters:
I20260924 17:39:30.306697 9653 leader_election.cc:288] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:30.306737 9658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:30.306919 9658 raft_consensus.cc:686] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [term 1 LEADER]: Becoming Leader. State: Replica: c04032e9504f499ca52c702d5dd5ccf4, State: Running, Role: LEADER
I20260924 17:39:30.307052 9653 sys_catalog.cc:564] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:30.307096 9658 consensus_queue.cc:237] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } }
I20260924 17:39:30.307468 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c04032e9504f499ca52c702d5dd5ccf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } } }
I20260924 17:39:30.307591 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:30.307653 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c04032e9504f499ca52c702d5dd5ccf4. Latest consensus state: current_term: 1 leader_uuid: "c04032e9504f499ca52c702d5dd5ccf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04032e9504f499ca52c702d5dd5ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33449 } } }
I20260924 17:39:30.307761 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:30.307976 9666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:30.308063 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.302449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.0.98.62:33449)
0924 17:39:30.302676 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.302694 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.302721 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:39:30.303316 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.303325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.303392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.303529 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.303762 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.304740 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.305874 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.306286 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.306393 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.307229 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:30.307233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:30.307233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:30.307458 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:30.307501 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:30.307523 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:30.307611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:30.307772 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 17:39:30.307857 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260924 17:39:30.308481 9666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:30.310277 9666 catalog_manager.cc:1347] Generated new cluster ID: 748d034c5edc4bec8b1fb3b0e7547f21
I20260924 17:39:30.310343 9666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:30.318982 9666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:30.319550 9666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:30.321499 9666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c04032e9504f499ca52c702d5dd5ccf4: Generated new TSK 0
I20260924 17:39:30.321698 9666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:30.325788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--builtin_ntp_servers=127.0.98.20:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 17:39:30.414816 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.415167 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.415212 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.416831 9677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.416927 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.416978 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:30.418484 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.418756 9677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.419134 9677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:30.424188 9685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.424149 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.424293 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.424801 9677 server_base.cc:1034] running on GCE node
I20260924 17:39:30.425010 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:30.425279 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:30.426469 9677 hybrid_clock.cc:648] HybridClock initialized: now 1790271570426446 us; error 41 us; skew 500 ppm
I20260924 17:39:30.427886 9677 webserver.cc:460] Webserver started at http://127.0.98.1:37063/ using document root <none> and password file <none>
I20260924 17:39:30.428143 9677 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:30.428207 9677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:30.428349 9677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:30.429749 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee1a5840d1044235a9d353a7feba31e1"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "e39ed72b6a723e5eda26dc6c739cbb6c"
server_key_iv: "339835783ff06743cb906e1936a661e8"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.430166 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee1a5840d1044235a9d353a7feba31e1"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "e39ed72b6a723e5eda26dc6c739cbb6c"
server_key_iv: "339835783ff06743cb906e1936a661e8"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.431905 9677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 17:39:30.432816 9691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.433037 9677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:30.433151 9677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "ee1a5840d1044235a9d353a7feba31e1"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "e39ed72b6a723e5eda26dc6c739cbb6c"
server_key_iv: "339835783ff06743cb906e1936a661e8"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.433238 9677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:30.449256 9677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:30.449613 9677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:30.449772 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:30.450016 9677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:30.450350 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:30.450407 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.450452 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:30.450476 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.456225 9677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36429
I20260924 17:39:30.456311 9804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36429 every 8 connection(s)
I20260924 17:39:30.456667 9677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:39:30.461527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9677
I20260924 17:39:30.461627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:30.461841 392 external_mini_cluster.cc:1428] Setting key c9b4fd0140581474f00cf64659b69146
I20260924 17:39:30.462335 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.457479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46835 (local address 127.0.98.62:33449)
0924 17:39:30.457524 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.457526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.457818 (+ 292us) server_negotiation.cc:407] Connection header received
0924 17:39:30.458376 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.458378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.458393 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.458453 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.459210 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.459721 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.460749 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.460952 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.461028 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.462029 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:30.462030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:30.462031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:30.462036 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:30.462061 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:30.462065 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:30.462126 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:30.462253 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 17:39:30.462277 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:30.462595 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.457478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33449 (local address 127.0.98.1:46835)
0924 17:39:30.457695 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:30.457731 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:30.458268 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:30.458525 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:30.458530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:30.458547 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:30.459110 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.459117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.459821 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.459822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.460643 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.460645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.461036 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.461037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.461181 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.461851 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:30.461851 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:30.461901 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:30.461933 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:30.462127 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:30.462134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:30.462137 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:30.462231 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 17:39:30.462343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260924 17:39:30.462889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--builtin_ntp_servers=127.0.98.20:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 17:39:30.463086 9805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33449
I20260924 17:39:30.463212 9805 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:30.463398 9805 heartbeater.cc:511] Master 127.0.98.62:33449 requested a full tablet report, sending...
I20260924 17:39:30.463905 9617 ts_manager.cc:194] Registered new tserver with Master: ee1a5840d1044235a9d353a7feba31e1 (127.0.98.1:36429)
I20260924 17:39:30.464530 9617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46835
W20260924 17:39:30.544320 9808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.544546 9808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.544569 9808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.545712 9808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.545750 9808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.545776 9808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:30.547117 9808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.547379 9808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.547652 9808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:39:30.552796 9808 server_base.cc:1034] running on GCE node
W20260924 17:39:30.552747 9814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.552778 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.552740 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.553324 9808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:30.553582 9808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:30.554787 9808 hybrid_clock.cc:648] HybridClock initialized: now 1790271570554726 us; error 48 us; skew 500 ppm
I20260924 17:39:30.556542 9808 webserver.cc:460] Webserver started at http://127.0.98.2:40685/ using document root <none> and password file <none>
I20260924 17:39:30.556867 9808 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:30.556941 9808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:30.557096 9808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:30.558457 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "069aaf04058f429db067634dcd589eba"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "900444dafbbb4db8397ab5aad5c4f07a"
server_key_iv: "a314123702689f357bd8f2d6fd2c5cff"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.558840 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "069aaf04058f429db067634dcd589eba"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "900444dafbbb4db8397ab5aad5c4f07a"
server_key_iv: "a314123702689f357bd8f2d6fd2c5cff"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.560240 9808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:30.561111 9822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.561298 9808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:30.561401 9808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "069aaf04058f429db067634dcd589eba"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "900444dafbbb4db8397ab5aad5c4f07a"
server_key_iv: "a314123702689f357bd8f2d6fd2c5cff"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.561492 9808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:30.571965 9808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:30.572293 9808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:30.572448 9808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:30.572674 9808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:30.573009 9808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:30.573063 9808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.573118 9808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:30.573143 9808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.579277 9808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45623
I20260924 17:39:30.579329 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45623 every 8 connection(s)
I20260924 17:39:30.579674 9808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:30.585003 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.580416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60091 (local address 127.0.98.62:33449)
0924 17:39:30.580459 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.580461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.580724 (+ 263us) server_negotiation.cc:407] Connection header received
0924 17:39:30.581276 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.581277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.581288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.581340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.582024 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.582522 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.583632 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.583825 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.583898 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.584703 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:30.584705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:30.584706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:30.584711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:30.584725 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:30.584730 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:30.584800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:30.584895 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:39:30.584922 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:39:30.585283 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.580411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33449 (local address 127.0.98.2:60091)
0924 17:39:30.580605 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:30.580639 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:30.581170 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:30.581430 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:30.581433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:30.581441 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:30.581933 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.581938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.582627 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.582628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.583535 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.583537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.583916 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.583918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.584054 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.584564 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:30.584564 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:30.584592 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:30.584614 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:30.584800 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:30.584807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:30.584810 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:30.584898 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:39:30.585003 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260924 17:39:30.585767 9936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33449
I20260924 17:39:30.585844 9936 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:30.586040 9936 heartbeater.cc:511] Master 127.0.98.62:33449 requested a full tablet report, sending...
I20260924 17:39:30.586434 9617 ts_manager.cc:194] Registered new tserver with Master: 069aaf04058f429db067634dcd589eba (127.0.98.2:45623)
I20260924 17:39:30.586793 9617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60091
I20260924 17:39:30.587925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9808
I20260924 17:39:30.588001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:30.588158 392 external_mini_cluster.cc:1428] Setting key ba2e6ef0d191679213509f80ffeeda50
I20260924 17:39:30.588991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--builtin_ntp_servers=127.0.98.20:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 17:39:30.670325 9939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.670572 9939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.670619 9939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.671923 9939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.671996 9939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.672041 9939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:30.673457 9939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.673739 9939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.674057 9939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:30.679370 9944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.679404 9947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.679666 9945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.679757 9939 server_base.cc:1034] running on GCE node
I20260924 17:39:30.680004 9939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:30.680306 9939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:30.681481 9939 hybrid_clock.cc:648] HybridClock initialized: now 1790271570681470 us; error 40 us; skew 500 ppm
I20260924 17:39:30.682899 9939 webserver.cc:460] Webserver started at http://127.0.98.3:40165/ using document root <none> and password file <none>
I20260924 17:39:30.683207 9939 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:30.683280 9939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:30.683439 9939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:30.684548 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "38805e178f644447863e6dc12493fa77"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "17f12597434775f834ab5712688fd91a"
server_key_iv: "fe60b11e9c68bbb18a608716ee4b03da"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.684922 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38805e178f644447863e6dc12493fa77"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "17f12597434775f834ab5712688fd91a"
server_key_iv: "fe60b11e9c68bbb18a608716ee4b03da"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.686280 9939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:30.687018 9953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.687256 9939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:30.687356 9939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "38805e178f644447863e6dc12493fa77"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "17f12597434775f834ab5712688fd91a"
server_key_iv: "fe60b11e9c68bbb18a608716ee4b03da"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.687444 9939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:30.699605 9939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:30.699935 9939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:30.700093 9939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:30.700331 9939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:30.700677 9939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:30.700732 9939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.700773 9939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:30.700802 9939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.706430 9939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35189
I20260924 17:39:30.706521 10066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35189 every 8 connection(s)
I20260924 17:39:30.706882 9939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:30.712504 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.707807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60273 (local address 127.0.98.62:33449)
0924 17:39:30.707899 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.707902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.708133 (+ 231us) server_negotiation.cc:407] Connection header received
0924 17:39:30.708688 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.708689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.708700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.708759 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.709473 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.709953 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.711133 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.711325 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.711396 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.712258 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:30.712260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:30.712260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:30.712265 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:30.712280 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:30.712284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:30.712334 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:30.712416 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:39:30.712440 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:39:30.712710 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.707809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33449 (local address 127.0.98.3:60273)
0924 17:39:30.708000 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:30.708042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:30.708574 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:30.708815 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:30.708817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:30.708827 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:30.709397 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.709402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.710059 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.710061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.710877 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:30.710909 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:30.711432 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:30.711434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:30.711567 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:30.712111 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:30.712112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:30.712142 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:30.712165 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:30.712346 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:30.712349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:30.712352 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:30.712405 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:39:30.712490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260924 17:39:30.713275 10067 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33449
I20260924 17:39:30.713361 10067 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:30.713548 10067 heartbeater.cc:511] Master 127.0.98.62:33449 requested a full tablet report, sending...
I20260924 17:39:30.713765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 9939
I20260924 17:39:30.713871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:30.713996 9617 ts_manager.cc:194] Registered new tserver with Master: 38805e178f644447863e6dc12493fa77 (127.0.98.3:35189)
I20260924 17:39:30.714073 392 external_mini_cluster.cc:1428] Setting key 3ddb0fbd696d5fd21e817d3842a5f330
I20260924 17:39:30.714438 9617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60273
I20260924 17:39:30.718986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:39:30.721388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9677
W20260924 17:39:30.721930 9657 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:30.720870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.98.62:33449)
0924 17:39:30.720962 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.720964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.720975 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:39:30.721044 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.721044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.721057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.721133 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.721850 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59644: BlockingRecv error: recv got EOF from 127.0.0.1:59644 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 17:39:30.726258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9808
I20260924 17:39:30.730692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9939
I20260924 17:39:30.735118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 9586
2026-09-24T17:39:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (595 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T17:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:30Z Disabled control of system clock
I20260924 17:39:30.761467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:39:30.842454 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.842664 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.842686 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.843881 10078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:30.843928 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.843941 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.843954 10078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:30.843967 10078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:30.845431 10078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.845679 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.845978 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:30.850644 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.850649 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:30.850607 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:30.850797 10078 server_base.cc:1034] running on GCE node
I20260924 17:39:30.851130 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:30.851428 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:30.852594 10078 hybrid_clock.cc:648] HybridClock initialized: now 1790271570852583 us; error 40 us; skew 500 ppm
I20260924 17:39:30.853916 10078 webserver.cc:460] Webserver started at http://127.0.98.62:40317/ using document root <none> and password file <none>
I20260924 17:39:30.854252 10078 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:30.854326 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:30.854483 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:30.855661 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "f9b4c133d11f4a7186f937f3b9795d03"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "1a263a2dbe23291aaf34197cdb3df63e"
server_key_iv: "e208fe297d06ce6b28fc0b1c51243bfd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.856029 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9b4c133d11f4a7186f937f3b9795d03"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "1a263a2dbe23291aaf34197cdb3df63e"
server_key_iv: "e208fe297d06ce6b28fc0b1c51243bfd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.857332 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:39:30.858054 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:30.858278 10078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:39:30.858379 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f9b4c133d11f4a7186f937f3b9795d03"
format_stamp: "Formatted at 2026-09-24 17:39:30 on dist-test-slave-d0pf"
server_key: "1a263a2dbe23291aaf34197cdb3df63e"
server_key_iv: "e208fe297d06ce6b28fc0b1c51243bfd"
server_key_version: "encryptionkey@0"
I20260924 17:39:30.858462 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:30.861886 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:30.862161 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:30.862308 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:30.866173 10078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46663
I20260924 17:39:30.866214 10144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46663 every 8 connection(s)
I20260924 17:39:30.866552 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:30.867269 10145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:30.869871 10145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03: Bootstrap starting.
I20260924 17:39:30.870396 10145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:30.870636 10145 log.cc:824] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:30.871383 10145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03: No bootstrap required, opened a new log
I20260924 17:39:30.872884 10145 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260924 17:39:30.873015 10145 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:30.873037 10145 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b4c133d11f4a7186f937f3b9795d03, State: Initialized, Role: FOLLOWER
I20260924 17:39:30.873173 10145 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260924 17:39:30.873256 10145 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:30.873279 10145 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:30.873306 10145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:30.873787 10145 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260924 17:39:30.873865 10145 leader_election.cc:302] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b4c133d11f4a7186f937f3b9795d03; no voters:
I20260924 17:39:30.873993 10145 leader_election.cc:288] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:30.874068 10148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:30.874245 10148 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [term 1 LEADER]: Becoming Leader. State: Replica: f9b4c133d11f4a7186f937f3b9795d03, State: Running, Role: LEADER
I20260924 17:39:30.874251 10145 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:30.874356 10148 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260924 17:39:30.874722 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b4c133d11f4a7186f937f3b9795d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260924 17:39:30.874811 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:30.874954 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b4c133d11f4a7186f937f3b9795d03. Latest consensus state: current_term: 1 leader_uuid: "f9b4c133d11f4a7186f937f3b9795d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b4c133d11f4a7186f937f3b9795d03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260924 17:39:30.875013 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03 [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:30.875548 10156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:30.875730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10078
I20260924 17:39:30.875816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:30.876039 392 external_mini_cluster.cc:1428] Setting key 300c100794090330851e3356f117dc14
I20260924 17:39:30.875986 10156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:30.877959 10156 catalog_manager.cc:1347] Generated new cluster ID: 2eceac3a1b6c4014820bb15519920d6f
I20260924 17:39:30.878014 10156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:30.878350 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:30.876691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.98.62:46663)
0924 17:39:30.876866 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:30.876873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:30.876895 (+ 22us) server_negotiation.cc:407] Connection header received
0924 17:39:30.876986 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:30.876992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:30.877033 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:30.877172 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:30.877845 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:30.877848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:30.877849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:30.877853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:30.877902 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:30.877913 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:30.877984 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:30.878174 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 17:39:30.878222 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260924 17:39:30.897125 10156 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:30.897320 10156 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:30.906138 10156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b4c133d11f4a7186f937f3b9795d03: Generated new TSK 0
I20260924 17:39:30.906366 10156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:39:30.910789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:39:30.992655 10169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:30.992897 10169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:30.992937 10169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:30.994139 10169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:30.994202 10169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:30.994244 10169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:30.995707 10169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:30.995970 10169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:30.996276 10169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:31.002161 10177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:31.002161 10175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:31.002202 10174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:31.002485 10169 server_base.cc:1034] running on GCE node
I20260924 17:39:31.002702 10169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:31.002979 10169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:31.004184 10169 hybrid_clock.cc:648] HybridClock initialized: now 1790271571004159 us; error 43 us; skew 500 ppm
I20260924 17:39:31.005620 10169 webserver.cc:460] Webserver started at http://127.0.98.1:43505/ using document root <none> and password file <none>
I20260924 17:39:31.005864 10169 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:31.005934 10169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:31.006071 10169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:31.007248 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddb363171f9d46588c78d45c28175cf7"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "619e1b739bff1bdcf300e7da9e02a1b0"
server_key_iv: "fac69fb5b2a0e210577fc9fae997e99e"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.007623 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddb363171f9d46588c78d45c28175cf7"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "619e1b739bff1bdcf300e7da9e02a1b0"
server_key_iv: "fac69fb5b2a0e210577fc9fae997e99e"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.009020 10169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:39:31.009826 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.010005 10169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:31.010118 10169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "ddb363171f9d46588c78d45c28175cf7"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "619e1b739bff1bdcf300e7da9e02a1b0"
server_key_iv: "fac69fb5b2a0e210577fc9fae997e99e"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.010206 10169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:31.014277 10169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:31.014583 10169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:31.014737 10169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:31.014964 10169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:31.015331 10169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:31.015389 10169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.015430 10169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:31.015457 10169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.021227 10169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39469
I20260924 17:39:31.021301 10296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39469 every 8 connection(s)
I20260924 17:39:31.021654 10169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:39:31.024667 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.022449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38339 (local address 127.0.98.62:46663)
0924 17:39:31.022518 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:31.022520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:31.022740 (+ 220us) server_negotiation.cc:407] Connection header received
0924 17:39:31.023278 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:31.023279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:31.023285 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:31.023339 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:31.024434 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:31.024435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:31.024435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:31.024439 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:31.024453 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:31.024457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:31.024521 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:31.024600 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 17:39:31.024613 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:39:31.024875 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.022425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.1:38339)
0924 17:39:31.022616 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:31.022652 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:31.023186 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:31.023402 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:31.023405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:31.023413 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:31.024277 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:31.024278 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:31.024307 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:31.024334 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:31.024531 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:31.024535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:31.024537 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:39:31.024594 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:39:31.024652 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260924 17:39:31.025353 10297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260924 17:39:31.025429 10297 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:31.025591 10297 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260924 17:39:31.025810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10169
I20260924 17:39:31.025903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:31.026068 392 external_mini_cluster.cc:1428] Setting key 4bb43159b1d531f6d92acdf0b4288b9a
I20260924 17:39:31.026087 10109 ts_manager.cc:194] Registered new tserver with Master: ddb363171f9d46588c78d45c28175cf7 (127.0.98.1:39469)
I20260924 17:39:31.026944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:39:31.107151 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:31.107388 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:31.107426 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:31.108630 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:31.108697 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:31.108740 10300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:31.110129 10300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:31.110406 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:31.110718 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:31.116839 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:31.116954 10300 server_base.cc:1034] running on GCE node
W20260924 17:39:31.116839 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:31.116843 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:31.117273 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:31.117554 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:31.118757 10300 hybrid_clock.cc:648] HybridClock initialized: now 1790271571118733 us; error 30 us; skew 500 ppm
I20260924 17:39:31.120298 10300 webserver.cc:460] Webserver started at http://127.0.98.2:36919/ using document root <none> and password file <none>
I20260924 17:39:31.120549 10300 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:31.120599 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:31.120765 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:31.121939 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f85682466214823a4f22cadf5aa33ce"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "b1cd412d2e6232f116ee6151f39a21b1"
server_key_iv: "abe284a88def4577fe170256fc7089a2"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.122303 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f85682466214823a4f22cadf5aa33ce"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "b1cd412d2e6232f116ee6151f39a21b1"
server_key_iv: "abe284a88def4577fe170256fc7089a2"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.123739 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260924 17:39:31.124552 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.124735 10300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:31.124841 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "2f85682466214823a4f22cadf5aa33ce"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "b1cd412d2e6232f116ee6151f39a21b1"
server_key_iv: "abe284a88def4577fe170256fc7089a2"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.124941 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:31.128465 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:31.128756 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:31.128895 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:31.129112 10300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:31.129417 10300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:31.129469 10300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.129509 10300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:31.129537 10300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.135217 10300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43977
I20260924 17:39:31.135260 10427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43977 every 8 connection(s)
I20260924 17:39:31.135592 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:31.138506 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.136366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42059 (local address 127.0.98.62:46663)
0924 17:39:31.136450 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:31.136452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:31.136660 (+ 208us) server_negotiation.cc:407] Connection header received
0924 17:39:31.137113 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:31.137114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:31.137119 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:31.137172 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:31.138279 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:31.138281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:31.138281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:31.138285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:31.138302 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:31.138306 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:31.138356 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:31.138440 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 17:39:31.138452 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:39:31.138701 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.136322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.2:42059)
0924 17:39:31.136532 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:31.136568 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:31.137031 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:31.137234 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:31.137237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:31.137246 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:31.138134 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:31.138135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:31.138163 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:31.138192 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:31.138365 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:31.138369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:31.138372 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:39:31.138442 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:39:31.138495 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260924 17:39:31.139183 10428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260924 17:39:31.139276 10428 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:31.139389 10428 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260924 17:39:31.139734 10109 ts_manager.cc:194] Registered new tserver with Master: 2f85682466214823a4f22cadf5aa33ce (127.0.98.2:43977)
I20260924 17:39:31.141599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10300
I20260924 17:39:31.141685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:31.141889 392 external_mini_cluster.cc:1428] Setting key 9be76b07044818db3cc44b7bd9b00b9b
I20260924 17:39:31.142714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:39:31.223953 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:31.224231 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:31.224273 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:31.225497 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:31.225576 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:31.225620 10431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:31.227010 10431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:31.227334 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:31.227682 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:31.232738 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:31.232810 10436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:31.232703 10437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:31.233335 10431 server_base.cc:1034] running on GCE node
I20260924 17:39:31.233603 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:31.233902 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:31.235101 10431 hybrid_clock.cc:648] HybridClock initialized: now 1790271571235086 us; error 26 us; skew 500 ppm
I20260924 17:39:31.236373 10431 webserver.cc:460] Webserver started at http://127.0.98.3:35233/ using document root <none> and password file <none>
I20260924 17:39:31.236651 10431 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:31.236725 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:31.236861 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:31.237949 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4dcd7d1466c46c3869f74a8d83b0c3b"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "633730bf279a6c67fc507e609ff520b6"
server_key_iv: "42db957951a0daeb8d7e0c8113f714b7"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.238335 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4dcd7d1466c46c3869f74a8d83b0c3b"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "633730bf279a6c67fc507e609ff520b6"
server_key_iv: "42db957951a0daeb8d7e0c8113f714b7"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.239734 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:39:31.240494 10445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.240700 10431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:39:31.240798 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "e4dcd7d1466c46c3869f74a8d83b0c3b"
format_stamp: "Formatted at 2026-09-24 17:39:31 on dist-test-slave-d0pf"
server_key: "633730bf279a6c67fc507e609ff520b6"
server_key_iv: "42db957951a0daeb8d7e0c8113f714b7"
server_key_version: "encryptionkey@0"
I20260924 17:39:31.240880 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:31.244305 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:31.244570 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:31.244714 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:31.244912 10431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:31.245234 10431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:31.245287 10431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.245326 10431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:31.245354 10431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:31.251677 10431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41703
I20260924 17:39:31.251713 10558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41703 every 8 connection(s)
I20260924 17:39:31.252076 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:31.255266 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.252859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34125 (local address 127.0.98.62:46663)
0924 17:39:31.252928 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:31.252930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:31.253175 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:39:31.253636 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:31.253637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:31.253642 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:31.253717 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:31.254895 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:31.254896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:31.254896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:39:31.254900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:31.254915 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:39:31.254919 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:31.255094 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:31.255196 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 17:39:31.255211 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:39:31.255455 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:31.252830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.3:34125)
0924 17:39:31.253038 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:31.253081 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:31.253550 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:31.253774 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:31.253778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:31.253786 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:31.254743 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:39:31.254743 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:31.254772 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:39:31.254802 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:31.255102 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:31.255110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:31.255117 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:39:31.255187 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:39:31.255240 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260924 17:39:31.255924 10559 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260924 17:39:31.256012 10559 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:31.256134 10559 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260924 17:39:31.256510 10109 ts_manager.cc:194] Registered new tserver with Master: e4dcd7d1466c46c3869f74a8d83b0c3b (127.0.98.3:41703)
I20260924 17:39:31.257243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10431
I20260924 17:39:31.257319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:31.257467 392 external_mini_cluster.cc:1428] Setting key 491d1a950db0464dd67a544ab5df0a9c
I20260924 17:39:31.258208 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 17:39:31.259972 10167 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:39:31.259503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.0.98.62:46663)
0924 17:39:31.259560 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:31.259562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:31.259704 (+ 142us) server_negotiation.cc:407] Connection header received
0924 17:39:31.259742 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:31.259743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:31.259746 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:31.259794 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:31.259899 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45062: BlockingRecv error: recv got EOF from 127.0.0.1:45062 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:39:31.260063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10169
I20260924 17:39:31.265395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10300
I20260924 17:39:31.269459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10431
I20260924 17:39:31.273617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10078
2026-09-24T17:39:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (538 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:31 dist-test-slave-d0pf krb5kdc[10569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:31 dist-test-slave-d0pf krb5kdc[10569](info): set up 2 sockets
Sep 24 17:39:31 dist-test-slave-d0pf krb5kdc[10569](info): commencing operation
krb5kdc: starting...
W20260924 17:39:32.316067 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:39:32.433995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:32.514577 10582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:32.514838 10582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:32.514878 10582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:32.516122 10582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:39:32.516201 10582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:32.516225 10582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:32.516258 10582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:39:32.516284 10582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:39:32.517795 10582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:32.518075 10582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:32.518307 10582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:32.523114 10590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:32.523110 10588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:32.523258 10582 server_base.cc:1034] running on GCE node
W20260924 17:39:32.523110 10587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:32.523609 10582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:32.523926 10582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:32.525072 10582 hybrid_clock.cc:648] HybridClock initialized: now 1790271572525047 us; error 37 us; skew 500 ppm
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:32.527551 10582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:39:32.528038 10582 webserver.cc:460] Webserver started at http://127.0.98.62:38745/ using document root <none> and password file <none>
I20260924 17:39:32.528285 10582 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:32.528368 10582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:32.528512 10582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:32.529568 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "036f27ff00ae473f839355a40459b39e"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "f833f69810e5004f6f487c067aea8bd3"
server_key_iv: "32e8e1b0fc1610879bcf0d7e97ddad5c"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.529932 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "036f27ff00ae473f839355a40459b39e"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "f833f69810e5004f6f487c067aea8bd3"
server_key_iv: "32e8e1b0fc1610879bcf0d7e97ddad5c"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.531354 10582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:39:32.532152 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.532338 10582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:32.532441 10582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "036f27ff00ae473f839355a40459b39e"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "f833f69810e5004f6f487c067aea8bd3"
server_key_iv: "32e8e1b0fc1610879bcf0d7e97ddad5c"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.532516 10582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:32.559513 10582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:32.561483 10582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:32.561657 10582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:32.566601 10582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34891
I20260924 17:39:32.566638 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34891 every 8 connection(s)
I20260924 17:39:32.567013 10582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:39:32.567992 10650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:32.568902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10582
I20260924 17:39:32.569001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:39:32.569186 392 external_mini_cluster.cc:1428] Setting key d219dcb23acf2a654562562c50c0a1f9
I20260924 17:39:32.571338 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: Bootstrap starting.
I20260924 17:39:32.571928 10650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:32.572225 10650 log.cc:824] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:32.572938 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: No bootstrap required, opened a new log
I20260924 17:39:32.575389 10650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } }
I20260924 17:39:32.575573 10650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:32.575609 10650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036f27ff00ae473f839355a40459b39e, State: Initialized, Role: FOLLOWER
I20260924 17:39:32.575747 10650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } }
I20260924 17:39:32.575845 10650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:39:32.575898 10650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:39:32.575946 10650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 0 FOLLOWER]: Advancing to term 1
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:32.576537 10650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } }
I20260924 17:39:32.576653 10650 leader_election.cc:302] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036f27ff00ae473f839355a40459b39e; no voters:
I20260924 17:39:32.576833 10650 leader_election.cc:288] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:39:32.576864 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:32.576967 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [term 1 LEADER]: Becoming Leader. State: Replica: 036f27ff00ae473f839355a40459b39e, State: Running, Role: LEADER
I20260924 17:39:32.577047 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } }
I20260924 17:39:32.577090 10650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:39:32.577790 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036f27ff00ae473f839355a40459b39e. Latest consensus state: current_term: 1 leader_uuid: "036f27ff00ae473f839355a40459b39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } } }
I20260924 17:39:32.577888 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:32.578208 10664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:39:32.578687 10664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:39:32.579566 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036f27ff00ae473f839355a40459b39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036f27ff00ae473f839355a40459b39e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34891 } } }
I20260924 17:39:32.579910 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e [sys.catalog]: This master's current role is: LEADER
I20260924 17:39:32.580435 10664 catalog_manager.cc:1347] Generated new cluster ID: 362db776993245ce85365e0ae79b5c9a
I20260924 17:39:32.580483 10664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:39:32.581013 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:32.569723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.98.62:34891)
0924 17:39:32.569886 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:32.569895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:32.569930 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:39:32.570336 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:32.570342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:32.570386 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:32.570495 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:32.571309 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.572295 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.573351 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.573742 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.573826 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:32.576705 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:32.576716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:32.576716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:32.576747 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:32.579009 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.579435 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.579438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.579438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.579545 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.579808 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.579809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.579810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.580110 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:32.580241 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:32.580674 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 17:39:32.580821 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260924 17:39:32.593567 10664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:39:32.594313 10664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:39:32.599507 10664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: Generated new TSK 0
I20260924 17:39:32.599696 10664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:39:32.634605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:32.715957 10676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:32.716270 10676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:32.716311 10676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:32.718498 10676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:32.718564 10676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:32.718611 10676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:39:32.720809 10676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:32.721071 10676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:32.721339 10676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:32.726963 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:32.727005 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:32.726981 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:32.727500 10676 server_base.cc:1034] running on GCE node
I20260924 17:39:32.727703 10676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:32.727965 10676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:32.729125 10676 hybrid_clock.cc:648] HybridClock initialized: now 1790271572729109 us; error 25 us; skew 500 ppm
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:32.731673 10676 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:39:32.732182 10676 webserver.cc:460] Webserver started at http://127.0.98.1:34145/ using document root <none> and password file <none>
I20260924 17:39:32.732419 10676 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:32.732489 10676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:32.732638 10676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:32.733698 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd0b349e190a4ef8b8557d3faab37fad"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "8dcee0061cfe5eb0d52cac18dfa334b2"
server_key_iv: "59f509bb168e433d4565d246c65b3924"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.734066 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd0b349e190a4ef8b8557d3faab37fad"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "8dcee0061cfe5eb0d52cac18dfa334b2"
server_key_iv: "59f509bb168e433d4565d246c65b3924"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.735461 10676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:39:32.736253 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.736438 10676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:32.736536 10676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "dd0b349e190a4ef8b8557d3faab37fad"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "8dcee0061cfe5eb0d52cac18dfa334b2"
server_key_iv: "59f509bb168e433d4565d246c65b3924"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.736621 10676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:32.747133 10676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:32.749455 10676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:32.749611 10676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:32.749838 10676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:32.750147 10676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:32.750201 10676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.750254 10676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:32.750283 10676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.756248 10676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40659
I20260924 17:39:32.756305 10804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40659 every 8 connection(s)
I20260924 17:39:32.756600 10676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:39:32.759816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10676
I20260924 17:39:32.759929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:39:32.760143 392 external_mini_cluster.cc:1428] Setting key a7e4ca2c36d4749aff068632f5891e98
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:32.766925 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:32.757281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51839 (local address 127.0.98.62:34891)
0924 17:39:32.757354 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:32.757356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:32.757571 (+ 215us) server_negotiation.cc:407] Connection header received
0924 17:39:32.758083 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:32.758085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:32.758097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:32.758152 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:32.759173 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.759710 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.761111 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.761344 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.761430 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:32.763885 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:32.763886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:32.763886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:32.763894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:32.765540 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.765948 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.765949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.765949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.766011 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.766266 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.766267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.766267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.766409 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:32.766489 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:32.766727 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 17:39:32.766823 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:39:32.766954 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:32.757281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34891 (local address 127.0.98.1:51839)
0924 17:39:32.757449 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:32.757485 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:32.757981 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:32.758231 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:32.758234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:32.758792 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:32.758988 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:32.758992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.759830 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.759832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:32.761015 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:32.761018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.761439 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.761440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:32.761579 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:32.762155 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:32.762168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:32.763787 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:32.765639 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:32.765645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:32.765646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:32.765862 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:32.766084 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:32.766084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:32.766085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:32.766192 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:32.766481 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:32.766486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:32.766601 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 17:39:32.766685 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:39:32.766800 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 17:39:32.767524 10805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34891
I20260924 17:39:32.767632 10805 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:32.767822 10805 heartbeater.cc:511] Master 127.0.98.62:34891 requested a full tablet report, sending...
I20260924 17:39:32.768312 10614 ts_manager.cc:194] Registered new tserver with Master: dd0b349e190a4ef8b8557d3faab37fad (127.0.98.1:40659)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:39:32.769017 10614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:39:32.785542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:32.865646 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:32.865870 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:32.865895 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:32.867102 10810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:32.867146 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:32.867172 10810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:39:32.868649 10810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:32.868902 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:32.869146 10810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:32.874269 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:32.874330 10810 server_base.cc:1034] running on GCE node
W20260924 17:39:32.874286 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:32.874313 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:32.874750 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:32.875082 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:32.876243 10810 hybrid_clock.cc:648] HybridClock initialized: now 1790271572876217 us; error 42 us; skew 500 ppm
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:32.878836 10810 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:39:32.879347 10810 webserver.cc:460] Webserver started at http://127.0.98.2:39751/ using document root <none> and password file <none>
I20260924 17:39:32.879575 10810 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:32.879657 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:32.879792 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:32.880841 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c15738606274629bc4778a5f256a56a"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "cd7f963536e94a9ca085ae04e29fd843"
server_key_iv: "202d895e5c87d4b312edcb2c9e61e7a1"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.881214 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c15738606274629bc4778a5f256a56a"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "cd7f963536e94a9ca085ae04e29fd843"
server_key_iv: "202d895e5c87d4b312edcb2c9e61e7a1"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.882546 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:39:32.883338 10825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.883543 10810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:32.883653 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "2c15738606274629bc4778a5f256a56a"
format_stamp: "Formatted at 2026-09-24 17:39:32 on dist-test-slave-d0pf"
server_key: "cd7f963536e94a9ca085ae04e29fd843"
server_key_iv: "202d895e5c87d4b312edcb2c9e61e7a1"
server_key_version: "encryptionkey@0"
I20260924 17:39:32.883726 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:32.900672 10810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:32.902583 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:32.902755 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:32.902994 10810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:32.903393 10810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:32.903451 10810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.903496 10810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:32.903525 10810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:32.909296 10810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37213
I20260924 17:39:32.909350 10938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37213 every 8 connection(s)
I20260924 17:39:32.909718 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:39:32.910146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10810
I20260924 17:39:32.910270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:39:32.910490 392 external_mini_cluster.cc:1428] Setting key e755bc1f1cc360b68aaf842ec8b5f269
Sep 24 17:39:32 dist-test-slave-d0pf krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:39:32.919984 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:32.910510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54775 (local address 127.0.98.62:34891)
0924 17:39:32.910560 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:32.910563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:32.910864 (+ 301us) server_negotiation.cc:407] Connection header received
0924 17:39:32.911519 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:32.911520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:32.911530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:32.911565 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:32.912917 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.913407 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.914456 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.914811 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.914894 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:32.917053 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:32.917054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:32.917055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:32.917063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:32.918607 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.919015 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.919016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.919016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.919118 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:32.919347 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:32.919348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:32.919348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:32.919453 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:32.919506 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:32.919802 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 17:39:32.919887 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:39:32.920186 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:32.910512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34891 (local address 127.0.98.2:54775)
0924 17:39:32.910695 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:32.910746 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:32.911416 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:32.911648 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:32.911653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:32.912501 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:32.912806 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:32.912814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.913505 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.913506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:32.914334 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:32.914336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:32.914896 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:32.914897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:32.914997 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:32.915606 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:32.915615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:32.916956 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:32.918709 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:32.918715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:32.918717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:32.918942 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:32.919193 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:32.919193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:32.919194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:32.919291 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:32.919507 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:32.919511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:32.919669 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 17:39:32.919787 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 17:39:32.919948 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260924 17:39:32.920682 10939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34891
I20260924 17:39:32.920766 10939 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:32.920970 10939 heartbeater.cc:511] Master 127.0.98.62:34891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 17:39:32.921396 10614 ts_manager.cc:194] Registered new tserver with Master: 2c15738606274629bc4778a5f256a56a (127.0.98.2:37213)
I20260924 17:39:32.921797 10614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:39:32.937377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:39:33.018517 10944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:39:33.018731 10944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:39:33.018754 10944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:39:33.019954 10944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:39:33.020000 10944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:39:33.020027 10944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:39:33.021449 10944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:39:33.021687 10944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:39:33.021924 10944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:39:33.027245 10950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:39:33.027294 10952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:33.027284 10944 server_base.cc:1034] running on GCE node
W20260924 17:39:33.027256 10949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:39:33.027653 10944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:39:33.027930 10944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:39:33.029076 10944 hybrid_clock.cc:648] HybridClock initialized: now 1790271573029062 us; error 28 us; skew 500 ppm
Sep 24 17:39:33 dist-test-slave-d0pf krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:39:33.031607 10944 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:39:33.032003 10944 webserver.cc:460] Webserver started at http://127.0.98.3:35509/ using document root <none> and password file <none>
I20260924 17:39:33.032222 10944 fs_manager.cc:359] Metadata directory not provided
I20260924 17:39:33.032263 10944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:39:33.032354 10944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:39:33.033368 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "b32b4af0d2a141eeb6fdc88001847dc5"
format_stamp: "Formatted at 2026-09-24 17:39:33 on dist-test-slave-d0pf"
server_key: "803438a20ff20b9ded09b143c58c3514"
server_key_iv: "e2dbc4da5ce1fba0e92c57f1692cf3bb"
server_key_version: "encryptionkey@0"
I20260924 17:39:33.033667 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b32b4af0d2a141eeb6fdc88001847dc5"
format_stamp: "Formatted at 2026-09-24 17:39:33 on dist-test-slave-d0pf"
server_key: "803438a20ff20b9ded09b143c58c3514"
server_key_iv: "e2dbc4da5ce1fba0e92c57f1692cf3bb"
server_key_version: "encryptionkey@0"
I20260924 17:39:33.035117 10944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:39:33.036021 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:33.036214 10944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:39:33.036298 10944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "b32b4af0d2a141eeb6fdc88001847dc5"
format_stamp: "Formatted at 2026-09-24 17:39:33 on dist-test-slave-d0pf"
server_key: "803438a20ff20b9ded09b143c58c3514"
server_key_iv: "e2dbc4da5ce1fba0e92c57f1692cf3bb"
server_key_version: "encryptionkey@0"
I20260924 17:39:33.036355 10944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:39:33.070343 10944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:39:33.072964 10944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:39:33.073125 10944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:39:33.073415 10944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:39:33.073722 10944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:39:33.073760 10944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:33.073784 10944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:39:33.073818 10944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:39:33.080056 10944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46477
I20260924 17:39:33.080139 11072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46477 every 8 connection(s)
I20260924 17:39:33.080487 10944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:39:33.082629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 10944
I20260924 17:39:33.082738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:39:33.082913 392 external_mini_cluster.cc:1428] Setting key aa1e128825d821b7c7239b69efa61f3e
Sep 24 17:39:33 dist-test-slave-d0pf krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:39:33.090220 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.081353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44277 (local address 127.0.98.62:34891)
0924 17:39:33.081397 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.081400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.081588 (+ 188us) server_negotiation.cc:407] Connection header received
0924 17:39:33.082102 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.082103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.082114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.082169 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:33.083304 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.083806 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.084913 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.085117 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.085254 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.087599 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:33.087601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:33.087601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:33.087609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:33.088848 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:33.089248 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:33.089249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:33.089250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:33.089313 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:33.089566 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:33.089567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:33.089568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:33.089689 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:33.089767 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.090038 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.090126 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:39:33.090425 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.081291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34891 (local address 127.0.98.3:44277)
0924 17:39:33.081476 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.081513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.082000 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.082227 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.082230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.082895 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:39:33.083209 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.083217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.083914 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.083914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.084804 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.084807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.085217 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.085218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.085447 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.085960 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:39:33.085969 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:39:33.087505 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:39:33.088944 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:33.088950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:33.088952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:33.089173 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:33.089380 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:39:33.089381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:39:33.089381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:39:33.089500 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:39:33.089758 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:39:33.089764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:39:33.089910 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.090019 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.090180 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260924 17:39:33.090925 11073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34891
I20260924 17:39:33.091020 11073 heartbeater.cc:464] Registering TS with master...
I20260924 17:39:33.091228 11073 heartbeater.cc:511] Master 127.0.98.62:34891 requested a full tablet report, sending...
I20260924 17:39:33.091634 10614 ts_manager.cc:194] Registered new tserver with Master: b32b4af0d2a141eeb6fdc88001847dc5 (127.0.98.3:46477)
I20260924 17:39:33.091976 10614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44277
I20260924 17:39:33.094743 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:39:33.102210 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.096126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.98.62:34891)
0924 17:39:33.096192 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.096194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.096281 (+ 87us) server_negotiation.cc:407] Connection header received
0924 17:39:33.096373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.096374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.096384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.096440 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:39:33.096980 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.097463 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.098184 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.098395 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.098478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.099758 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:39:33.099760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:39:33.099760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:39:33.099768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:39:33.101016 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:33.101387 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:33.101388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:33.101388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:33.101450 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:39:33.101640 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:39:33.101640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:39:33.101641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:39:33.101802 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:39:33.101911 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.102025 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.102106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:39:33.104000 10614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:39:33.105592 10614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:39:33.112874 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:58526)
0924 17:39:33.108597 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.108607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.108655 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.108984 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.108985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.108987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:33.109054 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.109056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110360 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.110361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.111546 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.111548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.112492 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.112493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.112592 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.112610 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.112691 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.112790 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":135,"thread_start_us":92,"threads_started":1}
I20260924 17:39:33.112865 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.98.1:40659)
0924 17:39:33.108812 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.108816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.108830 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:39:33.108866 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.108868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.108881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.109040 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:33.109123 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110257 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.111640 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.112405 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.112486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.112553 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.112719 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.112770 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":79,"threads_started":1}
I20260924 17:39:33.113061 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37213 (local address 127.0.0.1:43198)
0924 17:39:33.108536 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.108579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.108655 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.108965 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.108968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.108970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:33.109042 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.109044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110454 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.110455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.111747 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.111750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.112601 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.112601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.112657 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.112661 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.112711 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.112981 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":284,"thread_start_us":141,"threads_started":1}
I20260924 17:39:33.113077 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.98.2:37213)
0924 17:39:33.108552 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.108557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.108739 (+ 182us) server_negotiation.cc:407] Connection header received
0924 17:39:33.108785 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.108788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.108804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.108880 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:33.109125 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110256 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.111839 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.112523 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.112589 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.112641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.112798 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.112845 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260924 17:39:33.113900 10739 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 (DEFAULT_TABLE table=test-table [id=779b5501734a452898eda1b80102b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:33.113940 10873 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 (DEFAULT_TABLE table=test-table [id=779b5501734a452898eda1b80102b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:33.114262 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd6bcddf7f4ea2912bf73b4272df39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:33.114285 10873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd6bcddf7f4ea2912bf73b4272df39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:33.117010 11090 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: Bootstrap starting.
I20260924 17:39:33.117278 11091 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Bootstrap starting.
I20260924 17:39:33.117741 11090 tablet_bootstrap.cc:654] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:33.117981 11091 tablet_bootstrap.cc:654] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:33.118048 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46477 (local address 127.0.0.1:57226)
0924 17:39:33.108735 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.108747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.108802 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.109338 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.109339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.109341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:33.109393 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.109395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110238 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.110239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.111747 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.111750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.117711 (+ 5961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.117713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.117804 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.117811 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.117884 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.117950 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":93,"thread_start_us":95,"threads_started":1}
I20260924 17:39:33.118428 11090 log.cc:824] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:33.118494 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.108842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.0.98.3:46477)
0924 17:39:33.109181 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.109185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.109195 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:33.109236 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.109239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.109253 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.109341 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:33.109461 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.110131 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.112702 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.117597 (+ 4895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.117697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.117762 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.118223 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.118299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260924 17:39:33.118842 11091 log.cc:824] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:33.119259 11007 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 (DEFAULT_TABLE table=test-table [id=779b5501734a452898eda1b80102b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:39:33.119560 11090 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: No bootstrap required, opened a new log
I20260924 17:39:33.119620 11007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd6bcddf7f4ea2912bf73b4272df39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:39:33.119637 11090 ts_tablet_manager.cc:1397] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:39:33.119760 11091 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: No bootstrap required, opened a new log
I20260924 17:39:33.119849 11091 ts_tablet_manager.cc:1397] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 17:39:33.121886 11090 raft_consensus.cc:348] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.122107 11090 raft_consensus.cc:374] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:33.122148 11090 raft_consensus.cc:729] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0b349e190a4ef8b8557d3faab37fad, State: Initialized, Role: FOLLOWER
I20260924 17:39:33.122328 11090 consensus_queue.cc:260] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.122255 11091 raft_consensus.cc:348] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.122421 11091 raft_consensus.cc:374] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:33.122464 11091 raft_consensus.cc:729] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15738606274629bc4778a5f256a56a, State: Initialized, Role: FOLLOWER
I20260924 17:39:33.122678 10805 heartbeater.cc:503] Master 127.0.98.62:34891 was elected leader, sending a full tablet report...
I20260924 17:39:33.122671 11091 consensus_queue.cc:260] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.122709 11094 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Bootstrap starting.
I20260924 17:39:33.123003 10939 heartbeater.cc:503] Master 127.0.98.62:34891 was elected leader, sending a full tablet report...
I20260924 17:39:33.122990 11091 ts_tablet_manager.cc:1428] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:39:33.122671 11090 ts_tablet_manager.cc:1428] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:39:33.123876 11094 tablet_bootstrap.cc:654] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Neither blocks nor log segments found. Creating new log.
I20260924 17:39:33.124495 11094 log.cc:824] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Log is configured to *not* fsync() on all Append() calls
I20260924 17:39:33.125242 11094 tablet_bootstrap.cc:492] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: No bootstrap required, opened a new log
I20260924 17:39:33.125341 11094 ts_tablet_manager.cc:1397] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 17:39:33.127107 11094 raft_consensus.cc:348] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.127405 11094 raft_consensus.cc:374] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:39:33.127516 11094 raft_consensus.cc:729] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32b4af0d2a141eeb6fdc88001847dc5, State: Initialized, Role: FOLLOWER
I20260924 17:39:33.127658 11094 consensus_queue.cc:260] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.127954 11094 ts_tablet_manager.cc:1428] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:39:33.127981 11073 heartbeater.cc:503] Master 127.0.98.62:34891 was elected leader, sending a full tablet report...
I20260924 17:39:33.153962 11096 raft_consensus.cc:482] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:39:33.154098 11096 raft_consensus.cc:504] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.155010 11096 leader_election.cc:288] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd0b349e190a4ef8b8557d3faab37fad (127.0.98.1:40659), b32b4af0d2a141eeb6fdc88001847dc5 (127.0.98.3:46477)
I20260924 17:39:33.158759 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.154904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42387 (local address 127.0.98.1:40659)
0924 17:39:33.154964 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.154966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.155138 (+ 172us) server_negotiation.cc:407] Connection header received
0924 17:39:33.155170 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.155171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.155182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.155276 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:33.155424 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.156177 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.157762 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.158371 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.158454 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.158466 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.158646 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.158686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:39:33.158761 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.154988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39477 (local address 127.0.98.3:46477)
0924 17:39:33.155052 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.155054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.155064 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:39:33.155092 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.155093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.155107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.155170 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:39:33.155404 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.156163 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.157767 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.158371 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.158454 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.158466 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.158642 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.158684 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:39:33.158818 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.154782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46477 (local address 127.0.98.2:39477)
0924 17:39:33.154894 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.154907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.154948 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.155196 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.155197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.155198 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:33.155306 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.155308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.156258 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.156259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.157633 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.157637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.158467 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.158468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.158561 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.158563 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.158620 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.158679 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":113}
I20260924 17:39:33.159088 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.154834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.98.2:42387)
0924 17:39:33.155062 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.155071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:39:33.155124 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:39:33.155291 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:39:33.155292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:39:33.155293 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:39:33.155348 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.155349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.156258 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.156259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.157633 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:39:33.157637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.158467 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.158468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:39:33.158556 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.158561 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:39:33.158620 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:39:33.158679 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":66,"thread_start_us":115,"threads_started":1}
I20260924 17:39:33.159250 11027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd6bcddf7f4ea2912bf73b4272df39" candidate_uuid: "2c15738606274629bc4778a5f256a56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" is_pre_election: true
I20260924 17:39:33.159435 11027 raft_consensus.cc:2454] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15738606274629bc4778a5f256a56a in term 0.
I20260924 17:39:33.159464 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd6bcddf7f4ea2912bf73b4272df39" candidate_uuid: "2c15738606274629bc4778a5f256a56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0b349e190a4ef8b8557d3faab37fad" is_pre_election: true
I20260924 17:39:33.159648 10759 raft_consensus.cc:2454] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15738606274629bc4778a5f256a56a in term 0.
I20260924 17:39:33.159709 10827 leader_election.cc:302] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15738606274629bc4778a5f256a56a, b32b4af0d2a141eeb6fdc88001847dc5; no voters:
I20260924 17:39:33.159852 11096 raft_consensus.cc:2781] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:39:33.159902 11096 raft_consensus.cc:482] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:39:33.159938 11096 raft_consensus.cc:3037] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 0 FOLLOWER]: Advancing to term 1
W20260924 17:39:33.160228 10940 tablet.cc:2369] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:39:33.160775 11096 raft_consensus.cc:504] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.160943 11096 leader_election.cc:288] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [CANDIDATE]: Term 1 election: Requested vote from peers dd0b349e190a4ef8b8557d3faab37fad (127.0.98.1:40659), b32b4af0d2a141eeb6fdc88001847dc5 (127.0.98.3:46477)
I20260924 17:39:33.161082 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd6bcddf7f4ea2912bf73b4272df39" candidate_uuid: "2c15738606274629bc4778a5f256a56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0b349e190a4ef8b8557d3faab37fad"
I20260924 17:39:33.161187 10759 raft_consensus.cc:3037] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:33.161935 10759 raft_consensus.cc:2454] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15738606274629bc4778a5f256a56a in term 1.
I20260924 17:39:33.162117 10829 leader_election.cc:302] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15738606274629bc4778a5f256a56a, dd0b349e190a4ef8b8557d3faab37fad; no voters:
I20260924 17:39:33.162128 11027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd6bcddf7f4ea2912bf73b4272df39" candidate_uuid: "2c15738606274629bc4778a5f256a56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32b4af0d2a141eeb6fdc88001847dc5"
I20260924 17:39:33.162222 11096 raft_consensus.cc:2781] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:39:33.162242 11027 raft_consensus.cc:3037] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:39:33.162397 11096 raft_consensus.cc:686] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 1 LEADER]: Becoming Leader. State: Replica: 2c15738606274629bc4778a5f256a56a, State: Running, Role: LEADER
I20260924 17:39:33.162533 11096 consensus_queue.cc:237] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } }
I20260924 17:39:33.163174 11027 raft_consensus.cc:2454] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15738606274629bc4778a5f256a56a in term 1.
I20260924 17:39:33.163470 10614 catalog_manager.cc:5495] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c15738606274629bc4778a5f256a56a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c15738606274629bc4778a5f256a56a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15738606274629bc4778a5f256a56a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:39:33.170882 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.168702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.0.98.2:37213)
0924 17:39:33.168769 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:39:33.168782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 17:39:33.168790 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:39:33.168820 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:39:33.168821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:39:33.168838 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:39:33.168896 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:39:33.168987 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.169444 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.170178 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:39:33.170362 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:39:33.170414 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:39:33.170589 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:39:33.170688 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:39:33.170748 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:39:33.170784 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 17:39:33.170802 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:39:33.172824 10759 raft_consensus.cc:1261] T 2cdd6bcddf7f4ea2912bf73b4272df39 P dd0b349e190a4ef8b8557d3faab37fad [term 1 FOLLOWER]: Refusing update from remote peer 2c15738606274629bc4778a5f256a56a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:33.172832 11027 raft_consensus.cc:1261] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 1 FOLLOWER]: Refusing update from remote peer 2c15738606274629bc4778a5f256a56a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:39:33.173202 11096 consensus_queue.cc:1035] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32b4af0d2a141eeb6fdc88001847dc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:33.173343 11100 consensus_queue.cc:1035] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0b349e190a4ef8b8557d3faab37fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:39:33.176004 11103 mvcc.cc:205] Tried to move back new op lower bound from 7332952363712376832 to 7332952363674222592. Current Snapshot: MvccSnapshot[applied={T|T < 7332952363712376832}]
I20260924 17:39:33.176261 11105 mvcc.cc:205] Tried to move back new op lower bound from 7332952363712376832 to 7332952363674222592. Current Snapshot: MvccSnapshot[applied={T|T < 7332952363712376832}]
I20260924 17:39:33.177116 11104 mvcc.cc:205] Tried to move back new op lower bound from 7332952363712376832 to 7332952363674222592. Current Snapshot: MvccSnapshot[applied={T|T < 7332952363712376832}]
I20260924 17:39:33.178844 10611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:33.178951 10611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:39:33.179808 10611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 036f27ff00ae473f839355a40459b39e: Sending DeleteTablet for 3 replicas of tablet 2cdd6bcddf7f4ea2912bf73b4272df39
I20260924 17:39:33.180176 10873 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43198
I20260924 17:39:33.180188 11007 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57226
I20260924 17:39:33.180414 11112 tablet_replica.cc:330] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: stopping tablet replica
I20260924 17:39:33.180433 10739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdd6bcddf7f4ea2912bf73b4272df39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:39:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58526
I20260924 17:39:33.180540 11112 raft_consensus.cc:2229] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:39:33.180650 11112 raft_consensus.cc:2258] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:33.180771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10676
I20260924 17:39:33.181613 11112 ts_tablet_manager.cc:1905] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:33.182834 11113 tablet_replica.cc:330] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: stopping tablet replica
I20260924 17:39:33.182931 11113 raft_consensus.cc:2229] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:39:33.183059 11113 raft_consensus.cc:2258] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:39:33.183537 11113 ts_tablet_manager.cc:1905] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:39:33.183751 11112 ts_tablet_manager.cc:1918] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:33.183844 11112 log.cc:1196] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/2cdd6bcddf7f4ea2912bf73b4272df39
I20260924 17:39:33.184122 11112 ts_tablet_manager.cc:1939] T 2cdd6bcddf7f4ea2912bf73b4272df39 P b32b4af0d2a141eeb6fdc88001847dc5: Deleting consensus metadata
I20260924 17:39:33.184706 10599 catalog_manager.cc:4841] TS b32b4af0d2a141eeb6fdc88001847dc5 (127.0.98.3:46477): tablet 2cdd6bcddf7f4ea2912bf73b4272df39 (table test-table [id=779b5501734a452898eda1b80102b3ba]) successfully deleted
I20260924 17:39:33.185168 11113 ts_tablet_manager.cc:1918] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:39:33.185247 11113 log.cc:1196] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/2cdd6bcddf7f4ea2912bf73b4272df39
I20260924 17:39:33.185544 11113 ts_tablet_manager.cc:1939] T 2cdd6bcddf7f4ea2912bf73b4272df39 P 2c15738606274629bc4778a5f256a56a: Deleting consensus metadata
I20260924 17:39:33.186221 10599 catalog_manager.cc:4841] TS 2c15738606274629bc4778a5f256a56a (127.0.98.2:37213): tablet 2cdd6bcddf7f4ea2912bf73b4272df39 (table test-table [id=779b5501734a452898eda1b80102b3ba]) successfully deleted
W20260924 17:39:33.187731 10601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40659 (error 108)
I20260924 17:39:33.187922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10810
I20260924 17:39:33.188949 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:39:33.188128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:58536)
0924 17:39:33.188874 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 17:39:33.188919 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723}
W20260924 17:39:33.189095 10601 catalog_manager.cc:4584] TS dd0b349e190a4ef8b8557d3faab37fad (127.0.98.1:40659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cdd6bcddf7f4ea2912bf73b4272df39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
I20260924 17:39:33.194389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10944
I20260924 17:39:33.200778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 10582
2026-09-24T17:39:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1928 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:39:33 dist-test-slave-d0pf krb5kdc[11118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:39:33 dist-test-slave-d0pf krb5kdc[11118](info): set up 2 sockets
Sep 24 17:39:33 dist-test-slave-d0pf krb5kdc[11118](info): commencing operation
krb5kdc: starting...
W20260924 17:39:34.256117 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:39:34 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:39:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 17:39:34.417639 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcbbHA6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/postgres -l logfile start
2026-09-24 17:39:37.047 UTC [11148] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 17:39:37.047 UTC [11148] LOG: listening on IPv4 address "127.0.98.20", port 33137
2026-09-24 17:39:37.050 UTC [11148] LOG: listening on Unix socket "/tmp/.s.PGSQL.33137"
2026-09-24 17:39:37.073 UTC [11151] LOG: database system was shut down at 2026-09-24 17:39:35 UTC
2026-09-24 17:39:37.077 UTC [11148] LOG: database system is ready to accept connections
I20260924 17:39:38.044051 392 mini_postgres.cc:96] Postgres bound to 33137
2026-09-24 17:39:38.050 UTC [11159] FATAL: database "slave" does not exist
127.0.98.20:33137 - accepting connections
I20260924 17:39:38.050772 392 mini_ranger.cc:161] Starting Ranger...
I20260924 17:39:38.065603 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 17:39:38.541144 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 17:39:38.541256 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin
2026-09-24 17:39:38,829 [I] DB FLAVOR :POSTGRES
2026-09-24 17:39:38,830 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:39:38,830 [I] Checking connection..
2026-09-24 17:39:38,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 17:39:39,098 [I] Checking connection passed.
2026-09-24 17:39:39,098 [I] --------- Verifying version history table ---------
2026-09-24 17:39:39,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:39:39,365 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:39:39,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:39:39,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:39:39,620 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 17:39:39,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 17:39:39.887 UTC [11223] WARNING: there is no transaction in progress
2026-09-24 17:39:39,893 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 17:39:39,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:39:40,138 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 17:39:40,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:39:40,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 17:39:40,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 17:39:40,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d0pf.c.gcp-upstream.internal','N') ;"
2026-09-24 17:39:40,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:39:41,169 [I] Table x_portal_user does not exist in database ranger
2026-09-24 17:39:41,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:39:41,435 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 17:39:41,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:39:41,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:39:41,922 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 17:39:41,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 17:39:42.882 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:42.897 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:42.912 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:42.928 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:42.944 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:42.962 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.019 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.028 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.043 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.054 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.065 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.079 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.090 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.097 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.103 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43.110 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 17:39:43,736 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 17:39:43,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 17:39:43,959 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 17:39:43,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:39:44,216 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 17:39:44,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:39:44,462 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 17:39:44,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:39:44,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:39:44,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d0pf.c.gcp-upstream.internal';"
2026-09-24 17:39:45,190 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 17:39:45,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 17:39:45,452 [I] DB_PATCHES have already been applied
I20260924 17:39:45.457700 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcbbHA6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 17:39:45.457767 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 17:39:45.460839 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38553
Sep 24, 2026 5:39:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:39:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:39:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:39:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:39:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 17:39:46.578198 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.527s user 0.003s sys 0.006s
Sep 24, 2026 5:40:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:40:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 5:40:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:40:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:40:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 17:40:08.951082 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 17:40:09 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271609, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 17:40:09.050992 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 17:40:09.060883 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 17:40:09.958752 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 17:40:09.958855 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms
I20260924 17:40:09.958876 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33137
2026-09-24 17:40:10,134 [I] DB FLAVOR :POSTGRES
2026-09-24 17:40:10,135 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:40:10,135 [I] Checking connection
2026-09-24 17:40:10,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 17:40:10,401 [I] connection success
2026-09-24 17:40:10,401 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 17:40:10,401 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 17:40:10,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 17:40:10,666 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 17:40:10,666 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:40:10,666 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 17:40:10,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33137/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 17:40:10,934 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 17:40:11.415305 392 mini_ranger_kms.cc:310] Created kms service
I20260924 17:40:11.626924 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 17:40:11.669960 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 17:40:11 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271609, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:12.085460 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 17:40:12.085580 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 17:40:12.085604 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 17:40:12.089308 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11511
I20260924 17:40:12.089442 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45385
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 17:40:14.313673 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.263s user 0.019s sys 0.139s
I20260924 17:40:14.313791 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 17:40:14.313877 392 mini_ranger_kms.cc:401] 127.0.98.20:45385/kms/v1/keys
Sep 24 17:40:14 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 17:40:14 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 17:40:15 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 17:40:17 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:40:17.733836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41365
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45385/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:40:17.818472 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:40:17.818755 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:40:17.818797 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:40:17.820181 11562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:40:17.820262 11562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:40:17.820287 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:40:17.820310 11562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:40:17.820338 11562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:40:17.822690 11562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45385/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41365
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:40:17.823204 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:40:17.823473 11562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:40:17.828523 11567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:40:17.828523 11570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:17.828776 11562 server_base.cc:1034] running on GCE node
W20260924 17:40:17.828517 11568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:17.829099 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:40:17.829413 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:40:17.830619 11562 hybrid_clock.cc:648] HybridClock initialized: now 1790271617830614 us; error 39 us; skew 500 ppm
Sep 24 17:40:17 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:40:17.833317 11562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:40:17.833847 11562 webserver.cc:460] Webserver started at http://127.0.98.62:35797/ using document root <none> and password file <none>
I20260924 17:40:17.834084 11562 fs_manager.cc:359] Metadata directory not provided
I20260924 17:40:17.834143 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:40:17.834303 11562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:40:17 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:17.889945 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04"
format_stamp: "Formatted at 2026-09-24 17:40:17 on dist-test-slave-d0pf"
server_key: "b80405badb0de11558adbe360ea5bf84"
server_key_iv: "843205c98508c267e7789360ba868bf1"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:17.890429 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04"
format_stamp: "Formatted at 2026-09-24 17:40:17 on dist-test-slave-d0pf"
server_key: "b80405badb0de11558adbe360ea5bf84"
server_key_iv: "843205c98508c267e7789360ba868bf1"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:17.892542 11562 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 17:40:17.908285 11577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:17.908550 11562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:40:17.908667 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04"
format_stamp: "Formatted at 2026-09-24 17:40:17 on dist-test-slave-d0pf"
server_key: "b80405badb0de11558adbe360ea5bf84"
server_key_iv: "843205c98508c267e7789360ba868bf1"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:17.908777 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:40:17.933691 11562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:40:17.935846 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:40:17.936009 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:40:17.941323 11562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41365
I20260924 17:40:17.941412 11629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41365 every 8 connection(s)
I20260924 17:40:17.941792 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:40:17.942459 11630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:40:17.944855 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: Bootstrap starting.
I20260924 17:40:17.945473 11630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: Neither blocks nor log segments found. Creating new log.
I20260924 17:40:17.945755 11630 log.cc:824] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: Log is configured to *not* fsync() on all Append() calls
I20260924 17:40:17.946460 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: No bootstrap required, opened a new log
I20260924 17:40:17.948601 11630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } }
I20260924 17:40:17.948734 11630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:40:17.948757 11630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbc1ed1db544f87b8c5c37fa05d3f04, State: Initialized, Role: FOLLOWER
I20260924 17:40:17.948910 11630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } }
I20260924 17:40:17.948989 11630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:40:17.949012 11630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:40:17.949077 11630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:40:17.950001 11630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } }
I20260924 17:40:17.950125 11630 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbc1ed1db544f87b8c5c37fa05d3f04; no voters:
I20260924 17:40:17.950259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 11562
I20260924 17:40:17.950309 11630 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:40:17.950349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:40:17.950359 11633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:40:17.950505 11633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [term 1 LEADER]: Becoming Leader. State: Replica: 9dbc1ed1db544f87b8c5c37fa05d3f04, State: Running, Role: LEADER
I20260924 17:40:17.950534 392 external_mini_cluster.cc:1428] Setting key 922e2f90f127cb3f7287941c248f95ae
I20260924 17:40:17.950578 11630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:40:17.950583 11633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } }
I20260924 17:40:17.950910 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } } }
I20260924 17:40:17.950968 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:40:17.951087 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbc1ed1db544f87b8c5c37fa05d3f04. Latest consensus state: current_term: 1 leader_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc1ed1db544f87b8c5c37fa05d3f04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41365 } } }
I20260924 17:40:17.951126 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:40:17.951231 11638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:40:17.951732 11638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:40:17.952095 11630 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 17:40:17 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:40:17.963635 11638 catalog_manager.cc:1347] Generated new cluster ID: 24af2904bb3a46be98b1f779711bad1c
I20260924 17:40:17.963712 11638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:40:17.968220 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:17.951242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.98.62:41365)
0924 17:40:17.951871 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:17.951880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:17.951911 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:40:17.952030 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:17.952037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:17.952099 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:17.952231 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:40:17.954214 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:17.956940 (+ 2726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:17.958130 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:17.958783 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:17.958907 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:17.962138 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:40:17.962143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:40:17.962143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:40:17.962157 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:40:17.964888 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:17.965345 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:17.965348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:17.965348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:17.965454 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:17.966736 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:17.966737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:17.966737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:17.967158 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:40:17.967762 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:17.967851 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:40:17.967989 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":56,"threads_started":1}
I20260924 17:40:17.991178 11638 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:40:17.992453 11638 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:40:17.999846 11638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: Generated new TSK 0
I20260924 17:40:18.000169 11638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:40:18.726078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:40:18.845400 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:40:18.845620 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:40:18.845646 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:40:18.846860 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:40:18.846904 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:40:18.846932 11675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:40:18.848668 11675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:40:18.848945 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:40:18.849186 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:40:18.855257 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:40:18.855268 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:18.855571 11675 server_base.cc:1034] running on GCE node
W20260924 17:40:18.855966 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:18.856247 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:40:18.856518 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:40:18.857693 11675 hybrid_clock.cc:648] HybridClock initialized: now 1790271618857673 us; error 37 us; skew 500 ppm
Sep 24 17:40:18 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:40:18.860711 11675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:40:18.861368 11675 webserver.cc:460] Webserver started at http://127.0.98.1:37745/ using document root <none> and password file <none>
I20260924 17:40:18.861644 11675 fs_manager.cc:359] Metadata directory not provided
I20260924 17:40:18.861717 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:40:18.861868 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:40:18 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:18.874842 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "b643d5c40e9f4ec18b6b0df69a44ede6"
format_stamp: "Formatted at 2026-09-24 17:40:18 on dist-test-slave-d0pf"
server_key: "da1e50aa8e327eff50ac61f0102f1c8a"
server_key_iv: "deaec8405af961e9bac10fdb9903269c"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:18.875432 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b643d5c40e9f4ec18b6b0df69a44ede6"
format_stamp: "Formatted at 2026-09-24 17:40:18 on dist-test-slave-d0pf"
server_key: "da1e50aa8e327eff50ac61f0102f1c8a"
server_key_iv: "deaec8405af961e9bac10fdb9903269c"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:18.876883 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:40:18.890764 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:18.891062 11675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:40:18.891181 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "b643d5c40e9f4ec18b6b0df69a44ede6"
format_stamp: "Formatted at 2026-09-24 17:40:18 on dist-test-slave-d0pf"
server_key: "da1e50aa8e327eff50ac61f0102f1c8a"
server_key_iv: "deaec8405af961e9bac10fdb9903269c"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:18.891285 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:40:18.914170 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:40:18.916774 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:40:18.917014 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:40:18.917328 11675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:40:18.917758 11675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:40:18.917863 11675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:18.917956 11675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:40:18.918041 11675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:18.925407 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45795
I20260924 17:40:18.925928 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:40:18.926120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 11675
I20260924 17:40:18.926245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:40:18.926466 392 external_mini_cluster.cc:1428] Setting key f0347a80a41854d57a864bda3a0536a0
I20260924 17:40:18.927976 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45795 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 17:40:18 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 17:40:18.947911 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:18.929155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44273 (local address 127.0.98.62:41365)
0924 17:40:18.929299 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:18.929301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:18.936146 (+ 6845us) server_negotiation.cc:407] Connection header received
0924 17:40:18.936182 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:18.936183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:18.936197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:18.936262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:40:18.937075 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:18.937856 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:18.939427 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:18.939753 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:18.939814 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:18.944137 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:40:18.944138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:40:18.944139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:40:18.944147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:40:18.946182 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:18.946770 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:18.946771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:18.946771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:18.946858 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:18.947198 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:18.947199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:18.947199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:18.947326 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:40:18.947410 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:18.947725 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 17:40:18.947811 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260924 17:40:18.948109 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:18.929159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41365 (local address 127.0.98.1:44273)
0924 17:40:18.935221 (+ 6062us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:18.935294 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:18.935972 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:18.936400 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:18.936404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:18.936669 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:40:18.936950 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:18.936955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:18.937965 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:18.937966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:18.939330 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:18.939333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:18.939887 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:18.939888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:18.940047 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:18.940682 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:40:18.940693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:40:18.943999 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:40:18.946297 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:18.946306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:18.946309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:18.946669 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:18.946931 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:18.946932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:18.946932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:18.947110 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:18.947401 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:40:18.947407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:40:18.947582 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 17:40:18.947693 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:40:18.947860 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5996,"thread_start_us":83,"threads_started":1}
I20260924 17:40:18.948856 11806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41365
I20260924 17:40:18.948987 11806 heartbeater.cc:464] Registering TS with master...
I20260924 17:40:18.949204 11806 heartbeater.cc:511] Master 127.0.98.62:41365 requested a full tablet report, sending...
I20260924 17:40:18.949736 11594 ts_manager.cc:194] Registered new tserver with Master: b643d5c40e9f4ec18b6b0df69a44ede6 (127.0.98.1:45795)
I20260924 17:40:18.950445 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:40:18.963393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 17:40:19.084271 11812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:40:19.084568 11812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:40:19.084671 11812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:40:19.086427 11812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:40:19.086490 11812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:40:19.086532 11812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:40:19.088838 11812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:40:19.089107 11812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:40:19.089442 11812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:40:19.096369 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:40:19.096565 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:19.097749 11812 server_base.cc:1034] running on GCE node
W20260924 17:40:19.097976 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:19.098258 11812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:40:19.098549 11812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:40:19.101814 11812 hybrid_clock.cc:648] HybridClock initialized: now 1790271619101774 us; error 48 us; skew 500 ppm
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:40:19.105314 11812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:40:19.105875 11812 webserver.cc:460] Webserver started at http://127.0.98.2:42835/ using document root <none> and password file <none>
I20260924 17:40:19.106170 11812 fs_manager.cc:359] Metadata directory not provided
I20260924 17:40:19.106220 11812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:40:19.106364 11812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:19.127417 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e98c2958d9e42fca6ae9e8850e3640a"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "4acd64c5b91ef1225a90af01b6ad9258"
server_key_iv: "58ba9d74fefee6c2690553b2be40e2c9"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.127877 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e98c2958d9e42fca6ae9e8850e3640a"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "4acd64c5b91ef1225a90af01b6ad9258"
server_key_iv: "58ba9d74fefee6c2690553b2be40e2c9"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.131224 11812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 17:40:19.143935 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.144170 11812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:40:19.144248 11812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "0e98c2958d9e42fca6ae9e8850e3640a"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "4acd64c5b91ef1225a90af01b6ad9258"
server_key_iv: "58ba9d74fefee6c2690553b2be40e2c9"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.144335 11812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:40:19.186046 11812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:40:19.188604 11812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:40:19.188761 11812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:40:19.189002 11812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:40:19.189379 11812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:40:19.189436 11812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.189476 11812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:40:19.189546 11812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.197347 11812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46065
I20260924 17:40:19.197824 11812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:40:19.198734 11943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46065 every 8 connection(s)
I20260924 17:40:19.200310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 11812
I20260924 17:40:19.200438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:40:19.200621 392 external_mini_cluster.cc:1428] Setting key 60e74eef9334db0870ba852b9c87b872
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:40:19.210291 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.198934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55915 (local address 127.0.98.62:41365)
0924 17:40:19.198977 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.198979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.198990 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:40:19.199556 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.199558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.199571 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.199649 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:40:19.200263 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.201029 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.204622 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.204852 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.204944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.207291 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:40:19.207292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:40:19.207293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:40:19.207301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:40:19.208854 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.209276 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.209281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.209282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.209334 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.209603 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.209604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.209604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.209722 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:40:19.209799 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.210098 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.210195 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:40:19.210481 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.198644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41365 (local address 127.0.98.2:55915)
0924 17:40:19.198804 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.198851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.199423 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.199706 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.199709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.199904 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:40:19.200146 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.200157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.201134 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.201135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.204462 (+ 3327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.204466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.204965 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.204966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.205107 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.205746 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:40:19.205755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:40:19.207182 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:40:19.208946 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:19.208954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:19.208957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:19.209191 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:19.209410 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:19.209411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:19.209411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:19.209532 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:19.209789 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:40:19.209796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:40:19.209962 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.210067 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.210222 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 17:40:19.211088 11944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41365
I20260924 17:40:19.211213 11944 heartbeater.cc:464] Registering TS with master...
I20260924 17:40:19.211405 11944 heartbeater.cc:511] Master 127.0.98.62:41365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 17:40:19.211825 11594 ts_manager.cc:194] Registered new tserver with Master: 0e98c2958d9e42fca6ae9e8850e3640a (127.0.98.2:46065)
I20260924 17:40:19.212196 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:40:19.232788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:40:19.357856 11949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:40:19.358237 11949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:40:19.358300 11949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:40:19.360666 11949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:40:19.360740 11949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:40:19.360785 11949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:40:19.363224 11949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:40:19.363507 11949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:40:19.363852 11949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:40:19.370757 11955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:40:19.370771 11957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:19.370944 11949 server_base.cc:1034] running on GCE node
W20260924 17:40:19.371104 11954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:40:19.371317 11949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:40:19.371605 11949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:40:19.372802 11949 hybrid_clock.cc:648] HybridClock initialized: now 1790271619372785 us; error 34 us; skew 500 ppm
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:40:19.386878 11949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:40:19.387483 11949 webserver.cc:460] Webserver started at http://127.0.98.3:34157/ using document root <none> and password file <none>
I20260924 17:40:19.387753 11949 fs_manager.cc:359] Metadata directory not provided
I20260924 17:40:19.387826 11949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:40:19.387975 11949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:19.416877 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4324ef096cc45c884433be3da7a999c"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "c494c6178e98bffaa856ae40e16385de"
server_key_iv: "17a7552231cd9dfe62013bb9a01b1e99"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.417340 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4324ef096cc45c884433be3da7a999c"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "c494c6178e98bffaa856ae40e16385de"
server_key_iv: "17a7552231cd9dfe62013bb9a01b1e99"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.419247 11949 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260924 17:40:19.431967 11964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.433341 11949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 17:40:19.433501 11949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "c4324ef096cc45c884433be3da7a999c"
format_stamp: "Formatted at 2026-09-24 17:40:19 on dist-test-slave-d0pf"
server_key: "c494c6178e98bffaa856ae40e16385de"
server_key_iv: "17a7552231cd9dfe62013bb9a01b1e99"
server_key_version: "kuduclusterkey@0"
I20260924 17:40:19.433611 11949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:40:19.469084 11949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:40:19.475755 11949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:40:19.475999 11949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:40:19.476257 11949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:40:19.476655 11949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:40:19.476742 11949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.476811 11949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:40:19.476867 11949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:40:19.484274 11949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35627
I20260924 17:40:19.484817 11949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:40:19.485351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 11949
I20260924 17:40:19.485452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:40:19.485630 392 external_mini_cluster.cc:1428] Setting key eebeec3da4b295d0827c846acb49aff4
I20260924 17:40:19.486806 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35627 every 8 connection(s)
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:40:19.499737 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.488291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47941 (local address 127.0.98.62:41365)
0924 17:40:19.488335 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.488337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.488670 (+ 333us) server_negotiation.cc:407] Connection header received
0924 17:40:19.489523 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.489525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.489537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.489579 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:40:19.490464 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.491279 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.492812 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.493126 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.493167 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.495961 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:40:19.495962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:40:19.495963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:40:19.495971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:40:19.497960 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.498600 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.498601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.498601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.498672 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.498955 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.498956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.498956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.499116 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:40:19.499159 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.499552 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.499637 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:40:19.500528 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.488005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41365 (local address 127.0.98.3:47941)
0924 17:40:19.488543 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.488589 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.489395 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.489771 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.489775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.490041 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:40:19.490351 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.490358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.491396 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.491397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.492710 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.492713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.493259 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.493260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.493407 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.494045 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:40:19.494056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:40:19.495829 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:40:19.498135 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:19.498143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:19.498146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:19.498509 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:19.498741 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:40:19.498741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:40:19.498741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:40:19.498871 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:40:19.499206 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:40:19.499213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:40:19.499395 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.500117 (+ 722us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.500290 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":89,"threads_started":1}
I20260924 17:40:19.501189 12078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41365
I20260924 17:40:19.501283 12078 heartbeater.cc:464] Registering TS with master...
I20260924 17:40:19.501470 12078 heartbeater.cc:511] Master 127.0.98.62:41365 requested a full tablet report, sending...
I20260924 17:40:19.501871 11594 ts_manager.cc:194] Registered new tserver with Master: c4324ef096cc45c884433be3da7a999c (127.0.98.3:35627)
I20260924 17:40:19.502395 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47941
I20260924 17:40:19.509089 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:40:19.521195 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.512923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.62:41365)
0924 17:40:19.512967 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.512969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.513346 (+ 377us) server_negotiation.cc:407] Connection header received
0924 17:40:19.513446 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.513447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.513458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.513486 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:40:19.514434 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.515216 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.516295 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.516553 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.516600 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.518207 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:40:19.518209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:40:19.518209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:40:19.518216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:40:19.519984 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.520367 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.520368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.520368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.520421 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:40:19.520599 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:40:19.520599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:40:19.520600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:40:19.520713 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:40:19.520782 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.521009 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.521086 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:40:19.523445 11594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:40:19.525225 11594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:40:19.535790 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.528812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45795 (local address 127.0.0.1:38710)
0924 17:40:19.528944 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.528983 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.529043 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.531661 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.531665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.531667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:40:19.531734 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.531736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.532975 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.532976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.534309 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.534311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.535514 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.535515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.535594 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.535605 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.535665 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.535710 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260924 17:40:19.536273 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.529386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.0.98.1:45795)
0924 17:40:19.531427 (+ 2041us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.531432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.531441 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:40:19.531485 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.531487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.531504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.531579 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:40:19.531817 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.532886 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.534400 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.535348 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.535425 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.535481 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.536161 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.536211 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2008,"thread_start_us":69,"threads_started":1}
I20260924 17:40:19.537098 11740 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5249582b304fc0b3ed0f53f05f6abc (DEFAULT_TABLE table=test-table [id=10093d57eb594a8bb3385fd600cad601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:40:19.537312 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.529056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35627 (local address 127.0.0.1:44942)
0924 17:40:19.531150 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.531161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.531214 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.533356 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.533356 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.533358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:40:19.533398 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.533399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.534621 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.534622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.535834 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.535836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.537098 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.537099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.537169 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.537174 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.537204 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.537239 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"thread_start_us":68,"threads_started":1}
I20260924 17:40:19.537484 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5249582b304fc0b3ed0f53f05f6abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:40:19.537994 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.529637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.98.3:35627)
0924 17:40:19.533201 (+ 3564us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.533205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.533214 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:40:19.533249 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.533251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.533266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.533464 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:40:19.533532 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.534532 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.535968 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.536999 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.537721 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.537801 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.537856 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.537896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3527,"thread_start_us":75,"threads_started":1}
I20260924 17:40:19.538616 11993 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5249582b304fc0b3ed0f53f05f6abc (DEFAULT_TABLE table=test-table [id=10093d57eb594a8bb3385fd600cad601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:40:19.538930 11993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5249582b304fc0b3ed0f53f05f6abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:40:19.541800 12097 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: Bootstrap starting.
I20260924 17:40:19.542867 12097 tablet_bootstrap.cc:654] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: Neither blocks nor log segments found. Creating new log.
I20260924 17:40:19.543709 12097 log.cc:824] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:40:19.546255 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.529250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46065 (local address 127.0.0.1:58526)
0924 17:40:19.531291 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.531298 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.531332 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.533018 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.533019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.533021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:40:19.533070 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.533072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.544448 (+ 11376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.544450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.545158 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.545160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.546015 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.546016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.546092 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.546097 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.546130 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.546175 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2029,"thread_start_us":75,"threads_started":1}
I20260924 17:40:19.547214 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.529363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.98.2:46065)
0924 17:40:19.531578 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.531583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.531598 (+ 15us) server_negotiation.cc:407] Connection header received
0924 17:40:19.531641 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.531643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.531659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.531757 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:40:19.542975 (+ 11218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.544347 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.545279 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.545891 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.546900 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.547001 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.547092 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.547132 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2171,"thread_start_us":90,"threads_started":1}
I20260924 17:40:19.547987 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5249582b304fc0b3ed0f53f05f6abc (DEFAULT_TABLE table=test-table [id=10093d57eb594a8bb3385fd600cad601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:40:19.548336 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5249582b304fc0b3ed0f53f05f6abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:40:19.551208 12096 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: Bootstrap starting.
I20260924 17:40:19.551577 12099 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: Bootstrap starting.
I20260924 17:40:19.552057 12096 tablet_bootstrap.cc:654] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: Neither blocks nor log segments found. Creating new log.
I20260924 17:40:19.552429 12096 log.cc:824] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:40:19.552685 12099 tablet_bootstrap.cc:654] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: Neither blocks nor log segments found. Creating new log.
I20260924 17:40:19.553026 12099 log.cc:824] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:40:19.553282 12096 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: No bootstrap required, opened a new log
I20260924 17:40:19.553359 12096 ts_tablet_manager.cc:1397] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:40:19.555662 12097 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: No bootstrap required, opened a new log
I20260924 17:40:19.555735 12097 ts_tablet_manager.cc:1397] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20260924 17:40:19.556018 12096 raft_consensus.cc:348] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.556191 12096 raft_consensus.cc:374] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:40:19.556227 12096 raft_consensus.cc:729] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b643d5c40e9f4ec18b6b0df69a44ede6, State: Initialized, Role: FOLLOWER
I20260924 17:40:19.556366 12096 consensus_queue.cc:260] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.556619 12096 ts_tablet_manager.cc:1428] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:40:19.556730 11806 heartbeater.cc:503] Master 127.0.98.62:41365 was elected leader, sending a full tablet report...
I20260924 17:40:19.558073 12097 raft_consensus.cc:348] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.558238 12097 raft_consensus.cc:374] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:40:19.558277 12097 raft_consensus.cc:729] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4324ef096cc45c884433be3da7a999c, State: Initialized, Role: FOLLOWER
I20260924 17:40:19.558648 12097 consensus_queue.cc:260] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.558959 12097 ts_tablet_manager.cc:1428] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:40:19.559120 12078 heartbeater.cc:503] Master 127.0.98.62:41365 was elected leader, sending a full tablet report...
I20260924 17:40:19.560345 12099 tablet_bootstrap.cc:492] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: No bootstrap required, opened a new log
I20260924 17:40:19.560431 12099 ts_tablet_manager.cc:1397] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 17:40:19.562327 12099 raft_consensus.cc:348] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.562660 12099 raft_consensus.cc:374] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:40:19.562741 12099 raft_consensus.cc:729] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e98c2958d9e42fca6ae9e8850e3640a, State: Initialized, Role: FOLLOWER
I20260924 17:40:19.562915 12099 consensus_queue.cc:260] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.563272 12099 ts_tablet_manager.cc:1428] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:40:19.563410 11944 heartbeater.cc:503] Master 127.0.98.62:41365 was elected leader, sending a full tablet report...
I20260924 17:40:19.569687 12103 raft_consensus.cc:482] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:40:19.569842 12103 raft_consensus.cc:504] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.574347 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.570354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45795 (local address 127.0.98.3:38283)
0924 17:40:19.570466 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.570477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.570515 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.570755 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.570756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.570757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:40:19.570991 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.570994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.571968 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.571969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.573141 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.573144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.574114 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.574115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.574191 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.574195 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.574239 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.574273 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 17:40:19.574772 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.570590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38283 (local address 127.0.98.1:45795)
0924 17:40:19.570627 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.570629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.570636 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:40:19.570660 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.570660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.570670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.570714 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:40:19.571105 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.571876 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.573412 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.574010 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.574074 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.574084 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.574665 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.574708 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:40:19.575004 11758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5249582b304fc0b3ed0f53f05f6abc" candidate_uuid: "c4324ef096cc45c884433be3da7a999c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" is_pre_election: true
I20260924 17:40:19.575160 11758 raft_consensus.cc:2454] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4324ef096cc45c884433be3da7a999c in term 0.
I20260924 17:40:19.575556 11968 leader_election.cc:302] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b643d5c40e9f4ec18b6b0df69a44ede6, c4324ef096cc45c884433be3da7a999c; no voters:
I20260924 17:40:19.579653 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.575975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46065 (local address 127.0.98.3:48513)
0924 17:40:19.576130 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.576140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:40:19.576194 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:40:19.576697 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:40:19.576698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:40:19.576700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:40:19.576732 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.576733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.577620 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.577621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.578688 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:40:19.578691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.579447 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.579448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:40:19.579506 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.579509 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:40:19.579554 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:40:19.579586 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260924 17:40:19.579967 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.576067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48513 (local address 127.0.98.2:46065)
0924 17:40:19.576106 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.576108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.576476 (+ 368us) server_negotiation.cc:407] Connection header received
0924 17:40:19.576503 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.576504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.576515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.576561 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:40:19.576801 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.577518 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.578781 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.579337 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.579398 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.579408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.579866 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.579906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:40:19.580401 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5249582b304fc0b3ed0f53f05f6abc" candidate_uuid: "c4324ef096cc45c884433be3da7a999c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" is_pre_election: true
I20260924 17:40:19.580549 11890 raft_consensus.cc:2454] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4324ef096cc45c884433be3da7a999c in term 0.
I20260924 17:40:19.580775 12103 leader_election.cc:288] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b643d5c40e9f4ec18b6b0df69a44ede6 (127.0.98.1:45795), 0e98c2958d9e42fca6ae9e8850e3640a (127.0.98.2:46065)
I20260924 17:40:19.580914 12103 raft_consensus.cc:2781] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:40:19.580994 12103 raft_consensus.cc:482] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:40:19.581063 12103 raft_consensus.cc:3037] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:40:19.581946 12103 raft_consensus.cc:504] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.582268 11758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5249582b304fc0b3ed0f53f05f6abc" candidate_uuid: "c4324ef096cc45c884433be3da7a999c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6"
I20260924 17:40:19.582347 11758 raft_consensus.cc:3037] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:40:19.583684 11758 raft_consensus.cc:2454] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4324ef096cc45c884433be3da7a999c in term 1.
Sep 24 17:40:19 dist-test-slave-d0pf krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:19.583911 11968 leader_election.cc:302] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b643d5c40e9f4ec18b6b0df69a44ede6, c4324ef096cc45c884433be3da7a999c; no voters:
I20260924 17:40:19.584009 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5249582b304fc0b3ed0f53f05f6abc" candidate_uuid: "c4324ef096cc45c884433be3da7a999c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e98c2958d9e42fca6ae9e8850e3640a"
I20260924 17:40:19.584079 11890 raft_consensus.cc:3037] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:40:19.584906 11890 raft_consensus.cc:2454] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4324ef096cc45c884433be3da7a999c in term 1.
I20260924 17:40:19.585172 12103 leader_election.cc:288] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [CANDIDATE]: Term 1 election: Requested vote from peers b643d5c40e9f4ec18b6b0df69a44ede6 (127.0.98.1:45795), 0e98c2958d9e42fca6ae9e8850e3640a (127.0.98.2:46065)
I20260924 17:40:19.585278 12103 raft_consensus.cc:2781] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:40:19.585350 12103 raft_consensus.cc:686] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [term 1 LEADER]: Becoming Leader. State: Replica: c4324ef096cc45c884433be3da7a999c, State: Running, Role: LEADER
I20260924 17:40:19.585494 12103 consensus_queue.cc:237] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } }
I20260924 17:40:19.586395 11594 catalog_manager.cc:5495] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c reported cstate change: term changed from 0 to 1, leader changed from <none> to c4324ef096cc45c884433be3da7a999c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c4324ef096cc45c884433be3da7a999c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4324ef096cc45c884433be3da7a999c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 } health_report { overall_health: UNKNOWN } } }
I20260924 17:40:19.593636 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.590240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.98.3:35627)
0924 17:40:19.590397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 17:40:19.590399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:40:19.590406 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:40:19.590445 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:40:19.590446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:40:19.590464 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:40:19.590522 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:40:19.590716 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.591490 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.592650 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:40:19.592956 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:40:19.593014 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:40:19.593206 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:40:19.593328 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:40:19.593396 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:40:19.593520 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 17:40:19.593549 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260924 17:40:19.595700 11758 raft_consensus.cc:1261] T 6c5249582b304fc0b3ed0f53f05f6abc P b643d5c40e9f4ec18b6b0df69a44ede6 [term 1 FOLLOWER]: Refusing update from remote peer c4324ef096cc45c884433be3da7a999c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:40:19.595928 11890 raft_consensus.cc:1261] T 6c5249582b304fc0b3ed0f53f05f6abc P 0e98c2958d9e42fca6ae9e8850e3640a [term 1 FOLLOWER]: Refusing update from remote peer c4324ef096cc45c884433be3da7a999c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:40:19.596202 12105 consensus_queue.cc:1035] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b643d5c40e9f4ec18b6b0df69a44ede6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:40:19.599635 12105 consensus_queue.cc:1035] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e98c2958d9e42fca6ae9e8850e3640a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:40:19.608445 12108 mvcc.cc:205] Tried to move back new op lower bound from 7332952553860689920 to 7332952553823076352. Current Snapshot: MvccSnapshot[applied={T|T < 7332952553860689920}]
I20260924 17:40:19.608646 12109 mvcc.cc:205] Tried to move back new op lower bound from 7332952553860689920 to 7332952553823076352. Current Snapshot: MvccSnapshot[applied={T|T < 7332952553860689920 or (T in {7332952553860689920})}]
I20260924 17:40:19.612643 11594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:40:19.612751 11594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:40:19.613678 11594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dbc1ed1db544f87b8c5c37fa05d3f04: Sending DeleteTablet for 3 replicas of tablet 6c5249582b304fc0b3ed0f53f05f6abc
I20260924 17:40:19.614042 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5249582b304fc0b3ed0f53f05f6abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:40:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58526
I20260924 17:40:19.614331 11993 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5249582b304fc0b3ed0f53f05f6abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:40:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44942
I20260924 17:40:19.615020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 11675
W20260924 17:40:19.622743 11581 connection.cc:535] client connection to 127.0.98.1:45795 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 17:40:19.622859 11581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:40:19.623553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 11812
I20260924 17:40:19.624709 12119 tablet_replica.cc:330] T 6c5249582b304fc0b3ed0f53f05f6abc P c4324ef096cc45c884433be3da7a999c: stopping tablet replica
I20260924 17:40:19.625346 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.625226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45795 (local address 127.0.0.1:38726)
0924 17:40:19.625279 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.625313 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 17:40:19.625574 11581 catalog_manager.cc:4584] TS b643d5c40e9f4ec18b6b0df69a44ede6 (127.0.98.1:45795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c5249582b304fc0b3ed0f53f05f6abc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45795: connect: Connection refused (error 111)
I20260924 17:40:19.629778 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.629678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46065 (local address 127.0.0.1:58542)
0924 17:40:19.629723 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.629746 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 17:40:19.631130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 11949
I20260924 17:40:19.636317 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:40:19.636187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35627 (local address 127.0.0.1:44970)
0924 17:40:19.636237 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:40:19.636278 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 17:40:19.636371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 11562
I20260924 17:40:19.642886 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 17:40:19.987691 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 17:40:19.987800 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 17:40:20.332526 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 17:40:20.332 UTC [11148] LOG: received smart shutdown request
2026-09-24 17:40:20.336 UTC [11148] LOG: background worker "logical replication launcher" (PID 11157) exited with exit code 1
2026-09-24 17:40:20.336 UTC [11152] LOG: shutting down
2026-09-24 17:40:20.517 UTC [11148] LOG: database system is shut down
2026-09-24T17:40:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47427 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:40:20 dist-test-slave-d0pf krb5kdc[12129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:40:20 dist-test-slave-d0pf krb5kdc[12129](info): set up 2 sockets
Sep 24 17:40:20 dist-test-slave-d0pf krb5kdc[12129](info): commencing operation
krb5kdc: starting...
W20260924 17:40:21.678906 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:40:21 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:40:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 17:40:21.845966 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcbbHA6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/postgres -l logfile start
2026-09-24 17:40:24.395 UTC [12159] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 17:40:24.395 UTC [12159] LOG: listening on IPv4 address "127.0.98.20", port 50107
2026-09-24 17:40:24.399 UTC [12159] LOG: listening on Unix socket "/tmp/.s.PGSQL.50107"
2026-09-24 17:40:24.422 UTC [12162] LOG: database system was shut down at 2026-09-24 17:40:22 UTC
2026-09-24 17:40:24.427 UTC [12159] LOG: database system is ready to accept connections
I20260924 17:40:25.422998 392 mini_postgres.cc:96] Postgres bound to 50107
2026-09-24 17:40:25.429 UTC [12170] FATAL: database "slave" does not exist
127.0.98.20:50107 - accepting connections
I20260924 17:40:25.430025 392 mini_ranger.cc:161] Starting Ranger...
I20260924 17:40:25.445251 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 17:40:26.025799 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 17:40:26.025889 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin
2026-09-24 17:40:26,274 [I] DB FLAVOR :POSTGRES
2026-09-24 17:40:26,275 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:40:26,275 [I] Checking connection..
2026-09-24 17:40:26,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 17:40:26,558 [I] Checking connection passed.
2026-09-24 17:40:26,559 [I] --------- Verifying version history table ---------
2026-09-24 17:40:26,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:40:26,822 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:40:26,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:40:27,062 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:40:27,062 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 17:40:27,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 17:40:27.300 UTC [12234] WARNING: there is no transaction in progress
2026-09-24 17:40:27,307 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 17:40:27,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:40:27,546 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 17:40:27,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:40:27,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 17:40:27,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 17:40:28,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d0pf.c.gcp-upstream.internal','N') ;"
2026-09-24 17:40:28,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:40:28,533 [I] Table x_portal_user does not exist in database ranger
2026-09-24 17:40:28,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:40:28,812 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 17:40:28,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:40:29,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:40:29,311 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 17:40:29,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 17:40:30.322 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.340 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.358 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.375 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.391 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.408 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.465 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.472 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.488 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.500 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.511 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.527 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.537 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.544 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.552 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:30.560 UTC [12342] WARNING: there is no transaction in progress
2026-09-24 17:40:31,248 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 17:40:31,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 17:40:31,501 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 17:40:31,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:40:31,764 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 17:40:31,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:40:32,030 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 17:40:32,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:40:32,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:40:32,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d0pf.c.gcp-upstream.internal';"
2026-09-24 17:40:32,740 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 17:40:32,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 17:40:32,999 [I] DB_PATCHES have already been applied
I20260924 17:40:33.004949 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcbbHA6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 17:40:33.005023 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 17:40:33.007973 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57759
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:40:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 17:40:34.136451 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.706s user 0.000s sys 0.009s
Sep 24, 2026 5:40:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:40:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 5:40:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:40:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:40:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 17:40:55.246776 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 17:40:55 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271655, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 17:40:55.332391 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 17:40:55.342147 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 17:40:56.240062 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 17:40:56.240149 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms
I20260924 17:40:56.240166 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50107
2026-09-24 17:40:56,401 [I] DB FLAVOR :POSTGRES
2026-09-24 17:40:56,402 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:40:56,402 [I] Checking connection
2026-09-24 17:40:56,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 17:40:56,640 [I] connection success
2026-09-24 17:40:56,641 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 17:40:56,641 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 17:40:56,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 17:40:56,892 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 17:40:56,893 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:40:56,893 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 17:40:56,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 17:40:57,158 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 17:40:57.646116 392 mini_ranger_kms.cc:310] Created kms service
I20260924 17:40:57.814574 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 17:40:57.859162 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 17:40:57 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271655, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:40:58.262974 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 17:40:58.263121 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcbbHA6/build/release/bin/postgresql.jar:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcbbHA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 17:40:58.263142 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 17:40:58.266366 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12522
I20260924 17:40:58.266460 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39193
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 17:41:00.529377 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.197s user 0.005s sys 0.140s
I20260924 17:41:00.529486 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 17:41:00.529532 392 mini_ranger_kms.cc:401] 127.0.98.20:39193/kms/v1/keys
Sep 24 17:41:00 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 17:41:01 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 17:41:01 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 17:41:03 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:41:03.672485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:03.754761 12573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:03.754981 12573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:03.755003 12573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:03.756201 12573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:03.756247 12573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:03.756263 12573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:03.756276 12573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:03.756289 12573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:03.758010 12573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39193/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:03.758265 12573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:03.758527 12573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:03.763826 12581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:03.764002 12573 server_base.cc:1034] running on GCE node
W20260924 17:41:03.763829 12578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:03.764012 12579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:03.764420 12573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:03.764793 12573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:03.765976 12573 hybrid_clock.cc:648] HybridClock initialized: now 1790271663765955 us; error 44 us; skew 500 ppm
Sep 24 17:41:03 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:03.768975 12573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:03.769547 12573 webserver.cc:460] Webserver started at http://127.0.98.62:41701/ using document root <none> and password file <none>
I20260924 17:41:03.769798 12573 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:03.769863 12573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:03.770018 12573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:41:03 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:03.825909 12573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "482335843e9e4f99b39b7567c7d2529c"
format_stamp: "Formatted at 2026-09-24 17:41:03 on dist-test-slave-d0pf"
server_key: "46050bce1f0d3b82f8fb9781c9525640"
server_key_iv: "6ab290b64d78388dd7938a454f47b5c6"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:03.826319 12573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "482335843e9e4f99b39b7567c7d2529c"
format_stamp: "Formatted at 2026-09-24 17:41:03 on dist-test-slave-d0pf"
server_key: "46050bce1f0d3b82f8fb9781c9525640"
server_key_iv: "6ab290b64d78388dd7938a454f47b5c6"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:03.827860 12573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260924 17:41:03.843472 12588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:03.843719 12573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:03.843838 12573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "482335843e9e4f99b39b7567c7d2529c"
format_stamp: "Formatted at 2026-09-24 17:41:03 on dist-test-slave-d0pf"
server_key: "46050bce1f0d3b82f8fb9781c9525640"
server_key_iv: "6ab290b64d78388dd7938a454f47b5c6"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:03.843962 12573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:03.867008 12573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:03.869246 12573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:03.869414 12573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:03.874312 12573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260924 17:41:03.874384 12640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260924 17:41:03.874758 12573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:03.875488 12641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:03.877729 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Bootstrap starting.
I20260924 17:41:03.878317 12641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:03.878618 12641 log.cc:824] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:03.879174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 12573
I20260924 17:41:03.879282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:41:03.879452 392 external_mini_cluster.cc:1428] Setting key 6c2f21e4352711a8d2d1bdabe3787c6a
I20260924 17:41:03.879470 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: No bootstrap required, opened a new log
I20260924 17:41:03.882376 12641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:03.882651 12641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:03.882727 12641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482335843e9e4f99b39b7567c7d2529c, State: Initialized, Role: FOLLOWER
I20260924 17:41:03.882890 12641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:03.882967 12641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:03.882997 12641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:03.883100 12641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:03.883858 12641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:03.883972 12641 leader_election.cc:302] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 482335843e9e4f99b39b7567c7d2529c; no voters:
I20260924 17:41:03.884167 12641 leader_election.cc:288] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:41:03.884213 12646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:03.884419 12646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 LEADER]: Becoming Leader. State: Replica: 482335843e9e4f99b39b7567c7d2529c, State: Running, Role: LEADER
I20260924 17:41:03.884481 12641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:03.884578 12646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:03.884951 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "482335843e9e4f99b39b7567c7d2529c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:41:03.885089 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:03.885265 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 482335843e9e4f99b39b7567c7d2529c. Latest consensus state: current_term: 1 leader_uuid: "482335843e9e4f99b39b7567c7d2529c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:41:03.885358 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:03.885497 12655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:03.886009 12655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:03.886307 12641 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 17:41:03 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:03.895751 12655 catalog_manager.cc:1347] Generated new cluster ID: ea68c470102142ed81c4ec72d8364c67
I20260924 17:41:03.895912 12655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:03.896143 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:03.879952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.0.98.62:45549)
0924 17:41:03.880225 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:03.880233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:03.880257 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:41:03.882359 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:03.882368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:03.882423 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:03.882615 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:03.883445 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:03.884646 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:03.885754 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:03.886340 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:03.887068 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:03.890453 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:03.890514 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:03.890515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:03.890662 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:03.893782 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:03.894252 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:03.894256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:03.894256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:03.894377 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:03.894602 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:03.894603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:03.894604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:03.894942 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:03.895673 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:03.895751 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 17:41:03.895911 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":68,"threads_started":1}
I20260924 17:41:03.923507 12655 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:41:03.924396 12655 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:03.945911 12655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Generated new TSK 0
I20260924 17:41:03.946259 12655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:41:04.670200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:04.818039 12686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:04.818359 12686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:04.818400 12686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:04.820227 12686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:04.820349 12686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:04.820416 12686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:04.822875 12686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:04.823212 12686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:04.823560 12686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:04.830426 12691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:04.830611 12692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:04.830813 12694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:04.831022 12686 server_base.cc:1034] running on GCE node
I20260924 17:41:04.831271 12686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:04.831614 12686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:04.832823 12686 hybrid_clock.cc:648] HybridClock initialized: now 1790271664832809 us; error 42 us; skew 500 ppm
Sep 24 17:41:04 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:04.836094 12686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:04.836690 12686 webserver.cc:460] Webserver started at http://127.0.98.1:42213/ using document root <none> and password file <none>
I20260924 17:41:04.837028 12686 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:04.837112 12686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:04.837311 12686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:41:04 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:04.853463 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c43254018dd41f0893a99317a6c7291"
format_stamp: "Formatted at 2026-09-24 17:41:04 on dist-test-slave-d0pf"
server_key: "8e0ba016c0038d923601ecc70c085541"
server_key_iv: "6b6e4d09883f2f51da42ad7e7328a83f"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:04.853925 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c43254018dd41f0893a99317a6c7291"
format_stamp: "Formatted at 2026-09-24 17:41:04 on dist-test-slave-d0pf"
server_key: "8e0ba016c0038d923601ecc70c085541"
server_key_iv: "6b6e4d09883f2f51da42ad7e7328a83f"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:04.867578 12686 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.000s
I20260924 17:41:04.885964 12703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:04.886158 12686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:04.886248 12686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "7c43254018dd41f0893a99317a6c7291"
format_stamp: "Formatted at 2026-09-24 17:41:04 on dist-test-slave-d0pf"
server_key: "8e0ba016c0038d923601ecc70c085541"
server_key_iv: "6b6e4d09883f2f51da42ad7e7328a83f"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:04.886355 12686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:04.897720 12686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:04.900352 12686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:04.900559 12686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:04.900874 12686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:04.901259 12686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:04.901341 12686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:04.901444 12686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:04.901513 12686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:04.910140 12686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40913
I20260924 17:41:04.910602 12686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:04.911978 12816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40913 every 8 connection(s)
Sep 24 17:41:04 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:04.919705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 12686
I20260924 17:41:04.919804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:41:04.920030 392 external_mini_cluster.cc:1428] Setting key a4218a3cea29a7b81c2bc6ed26227f6b
I20260924 17:41:04.924058 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:04.911558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:50367)
0924 17:41:04.912154 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:04.912195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:04.912921 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:04.913398 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:04.913404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:04.913658 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:04.913952 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:04.913959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:04.914977 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:04.914978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:04.916313 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:04.916316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:04.916855 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:04.916856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:04.917018 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:04.917652 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:04.917664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:04.919535 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:04.922135 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:04.922146 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:04.922149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:04.922524 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:04.922793 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:04.922794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:04.922794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:04.922948 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:04.923302 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:04.923308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:04.923517 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 17:41:04.923621 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 17:41:04.923797 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":79,"threads_started":1}
I20260924 17:41:04.924731 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:04.911788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50367 (local address 127.0.98.62:45549)
0924 17:41:04.913158 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:04.913160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:04.913168 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:04.913194 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:04.913194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:04.913207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:04.913281 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:04.914081 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:04.914864 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:04.916418 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:04.916743 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:04.916816 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:04.919684 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:04.919686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:04.919686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:04.919696 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:04.922024 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:04.922621 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:04.922623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:04.922623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:04.922696 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:04.923064 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:04.923065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:04.923066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:04.923211 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:04.924489 (+ 1278us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:04.924536 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 17:41:04.924628 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":71,"threads_started":1}
I20260924 17:41:04.925298 12817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:04.925407 12817 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:04.925595 12817 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:04.926208 12605 ts_manager.cc:194] Registered new tserver with Master: 7c43254018dd41f0893a99317a6c7291 (127.0.98.1:40913)
I20260924 17:41:04.927117 12605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50367
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:41:04.955484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:05.083868 12823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:05.084327 12823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:05.084438 12823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:05.086562 12823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:05.086685 12823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:05.086783 12823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:05.089111 12823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:05.089638 12823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:05.089972 12823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:05.096962 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.097354 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.097609 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:05.098802 12823 server_base.cc:1034] running on GCE node
I20260924 17:41:05.099058 12823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:05.099344 12823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:05.107266 12823 hybrid_clock.cc:648] HybridClock initialized: now 1790271665107222 us; error 39 us; skew 500 ppm
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:05.110949 12823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:05.112150 12823 webserver.cc:460] Webserver started at http://127.0.98.2:42415/ using document root <none> and password file <none>
I20260924 17:41:05.112506 12823 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:05.112622 12823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:05.112805 12823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:05.128628 12823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "62f859229f2f4ad4aa5e0239cc239049"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "288b7990c0af6411c9b58d6d2597e857"
server_key_iv: "9d1addc5f399b18f8478b6c3e1a1004d"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.129076 12823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62f859229f2f4ad4aa5e0239cc239049"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "288b7990c0af6411c9b58d6d2597e857"
server_key_iv: "9d1addc5f399b18f8478b6c3e1a1004d"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.135537 12823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 17:41:05.159484 12840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.160087 12823 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 17:41:05.160198 12823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "62f859229f2f4ad4aa5e0239cc239049"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "288b7990c0af6411c9b58d6d2597e857"
server_key_iv: "9d1addc5f399b18f8478b6c3e1a1004d"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.160292 12823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:05.181396 12823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:05.183990 12823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:05.184201 12823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:05.184513 12823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:05.184917 12823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:05.185006 12823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.185073 12823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:05.185127 12823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.191509 12823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37979
I20260924 17:41:05.192045 12823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:41:05.195235 12954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37979 every 8 connection(s)
I20260924 17:41:05.197448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 12823
I20260924 17:41:05.197556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:41:05.197770 392 external_mini_cluster.cc:1428] Setting key 02a153baea854e3be39fa7470fbdc27d
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:05.207130 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.196092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39151 (local address 127.0.98.62:45549)
0924 17:41:05.196260 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:05.196263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:05.196408 (+ 145us) server_negotiation.cc:407] Connection header received
0924 17:41:05.197234 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:05.197236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:05.197252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:05.197286 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:05.198258 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.199110 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.200546 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.200851 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.200893 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.203646 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:05.203648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:05.203648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:05.203657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:05.205403 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.205913 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.205914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.205915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.205973 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.206257 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.206258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.206258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.206397 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:05.206438 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:05.206880 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 17:41:05.206974 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260924 17:41:05.207908 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.196052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:39151)
0924 17:41:05.196242 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:05.196305 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:05.197094 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:05.197501 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:05.197506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:05.197810 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:05.198133 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.198141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.199224 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.199225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.200445 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.200447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.200995 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.200996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.201136 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.201840 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:05.201851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:05.203522 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:05.205501 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.205509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.205513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.205824 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.206044 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.206044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.206045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.206179 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.206483 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:05.206496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:05.206699 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 17:41:05.207502 (+ 803us) client_negotiation.cc:240] Negotiation successful
0924 17:41:05.207672 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260924 17:41:05.208640 12955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:05.208747 12955 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:05.208931 12955 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:05.209301 12605 ts_manager.cc:194] Registered new tserver with Master: 62f859229f2f4ad4aa5e0239cc239049 (127.0.98.2:37979)
I20260924 17:41:05.209758 12605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39151
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:41:05.242515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:05.374014 12960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:05.374437 12960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:05.374547 12960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:05.376668 12960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:05.376952 12960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:05.377075 12960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:05.379870 12960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:05.380419 12960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:05.380795 12960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:41:05.389223 12960 server_base.cc:1034] running on GCE node
W20260924 17:41:05.389163 12966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.389511 12965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.395247 12968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:05.395709 12960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:05.396090 12960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:05.399394 12960 hybrid_clock.cc:648] HybridClock initialized: now 1790271665399368 us; error 44 us; skew 500 ppm
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:05.403187 12960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:05.404193 12960 webserver.cc:460] Webserver started at http://127.0.98.3:34513/ using document root <none> and password file <none>
I20260924 17:41:05.404568 12960 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:05.404644 12960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:05.404781 12960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:05.427753 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d987cb3ae854e6b80952c8cbd0e86a6"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "502136f3e2cb2a70f4442c89cd3869d2"
server_key_iv: "545a8f3c290010f58498ee548828b5ab"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.430095 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d987cb3ae854e6b80952c8cbd0e86a6"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "502136f3e2cb2a70f4442c89cd3869d2"
server_key_iv: "545a8f3c290010f58498ee548828b5ab"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.441934 12960 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260924 17:41:05.456496 12975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.456825 12960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:05.456966 12960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "1d987cb3ae854e6b80952c8cbd0e86a6"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "502136f3e2cb2a70f4442c89cd3869d2"
server_key_iv: "545a8f3c290010f58498ee548828b5ab"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.457095 12960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:05.482218 12960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:05.484992 12960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:05.485214 12960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:05.485512 12960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:05.486032 12960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:05.486126 12960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.486202 12960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:05.486260 12960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.492964 12960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40049
I20260924 17:41:05.493496 12960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:05.496548 13088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40049 every 8 connection(s)
I20260924 17:41:05.502380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 12960
I20260924 17:41:05.502485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:41:05.502673 392 external_mini_cluster.cc:1428] Setting key 7a0b1cd9c8e1005ade6e06a3e71243f8
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:05.509080 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.497305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52383 (local address 127.0.98.62:45549)
0924 17:41:05.497352 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:05.497354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:05.497918 (+ 564us) server_negotiation.cc:407] Connection header received
0924 17:41:05.498650 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:05.498651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:05.498665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:05.498710 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:05.499645 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.500452 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.502191 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.502505 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.502554 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.505538 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:05.505540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:05.505541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:05.505548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:05.507319 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.507881 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.507883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.507883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.507946 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.508259 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.508260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.508260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.508415 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:05.508462 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:05.508893 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 17:41:05.508983 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:41:05.509904 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.497279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:52383)
0924 17:41:05.497756 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:05.497819 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:05.498515 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:05.498921 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:05.498926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:05.499224 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:05.499517 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.499524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.500575 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.500576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.502064 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.502067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.502819 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.502820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.502988 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.503679 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:05.503691 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:05.505416 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:05.507429 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.507437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.507440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.507780 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.508024 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.508024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.508025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.508171 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.508514 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:05.508521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:05.508725 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 17:41:05.509473 (+ 748us) client_negotiation.cc:240] Negotiation successful
0924 17:41:05.509666 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":105,"threads_started":1}
I20260924 17:41:05.510569 13089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:05.510680 13089 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:05.510879 13089 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:05.511323 12605 ts_manager.cc:194] Registered new tserver with Master: 1d987cb3ae854e6b80952c8cbd0e86a6 (127.0.98.3:40049)
I20260924 17:41:05.511834 12605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52383
I20260924 17:41:05.514392 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:05.515172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 12686
I20260924 17:41:05.521979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 12823
I20260924 17:41:05.528247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 12960
I20260924 17:41:05.535156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 12573
I20260924 17:41:05.540123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=41701
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:05.621953 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:05.622171 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:05.622195 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:05.623425 13092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:05.623471 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:05.623484 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:05.623497 13092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:05.623510 13092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:05.625226 13092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--ranger_config_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:05.625484 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:05.625715 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:05.630585 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:05.630713 13092 server_base.cc:1034] running on GCE node
W20260924 17:41:05.630602 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.630602 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:05.631085 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:05.631414 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:05.632586 13092 hybrid_clock.cc:648] HybridClock initialized: now 1790271665632572 us; error 44 us; skew 500 ppm
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:05.634961 13092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:05.635483 13092 webserver.cc:460] Webserver started at http://127.0.98.62:41701/ using document root <none> and password file <none>
I20260924 17:41:05.635700 13092 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:05.635776 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:05.690100 13092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:05.691136 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.691309 13092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:05.691428 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "482335843e9e4f99b39b7567c7d2529c"
format_stamp: "Formatted at 2026-09-24 17:41:03 on dist-test-slave-d0pf"
server_key: "46050bce1f0d3b82f8fb9781c9525640"
server_key_iv: "6ab290b64d78388dd7938a454f47b5c6"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.691810 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:05.701581 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:05.703719 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:05.703938 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:05.708675 13092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260924 17:41:05.708709 13160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260924 17:41:05.709149 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:05.710727 13161 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:41:05.711637 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Bootstrap starting.
I20260924 17:41:05.716420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13092
I20260924 17:41:05.716717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40913
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=42213
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:05.722517 13161 log.cc:824] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:05.725138 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:05.725521 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Bootstrap complete.
I20260924 17:41:05.729595 13161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:05.729964 13161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482335843e9e4f99b39b7567c7d2529c, State: Initialized, Role: FOLLOWER
I20260924 17:41:05.730139 13161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:05.730365 13161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:05.730464 13161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:05.730515 13161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:41:05.732050 13161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:05.732177 13161 leader_election.cc:302] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 482335843e9e4f99b39b7567c7d2529c; no voters:
I20260924 17:41:05.732385 13161 leader_election.cc:288] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:41:05.732487 13165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:41:05.732726 13165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [term 2 LEADER]: Becoming Leader. State: Replica: 482335843e9e4f99b39b7567c7d2529c, State: Running, Role: LEADER
I20260924 17:41:05.732777 13161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:05.734289 13165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 17:41:05.735073 13161 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 17:41:05.735118 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "482335843e9e4f99b39b7567c7d2529c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:41:05.735198 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:05.735337 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 482335843e9e4f99b39b7567c7d2529c. Latest consensus state: current_term: 2 leader_uuid: "482335843e9e4f99b39b7567c7d2529c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482335843e9e4f99b39b7567c7d2529c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 17:41:05.735378 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:05.735616 13179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:05.736166 13179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:05.736359 13179 catalog_manager.cc:1259] Loaded cluster ID: ea68c470102142ed81c4ec72d8364c67
I20260924 17:41:05.736441 13179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:05.737703 13179 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:05.740562 13179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Loaded TSK: 0
I20260924 17:41:05.741110 13179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:41:05.838788 13163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:05.839185 13163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:05.839289 13163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:05.841266 13163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:05.841408 13163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:05.841507 13163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:05.844008 13163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42213
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:05.844559 13163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:05.844913 13163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:05.851847 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.852226 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:05.852502 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:05.853753 13163 server_base.cc:1034] running on GCE node
I20260924 17:41:05.853974 13163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:05.854246 13163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:05.855398 13163 hybrid_clock.cc:648] HybridClock initialized: now 1790271665855376 us; error 34 us; skew 500 ppm
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:05.858675 13163 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:05.859236 13163 webserver.cc:460] Webserver started at http://127.0.98.1:42213/ using document root <none> and password file <none>
I20260924 17:41:05.859535 13163 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:05.859614 13163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:05.883440 13163 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 17:41:05.884335 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.884537 13163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:05.884618 13163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "7c43254018dd41f0893a99317a6c7291"
format_stamp: "Formatted at 2026-09-24 17:41:04 on dist-test-slave-d0pf"
server_key: "8e0ba016c0038d923601ecc70c085541"
server_key_iv: "6b6e4d09883f2f51da42ad7e7328a83f"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:05.885094 13163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:05.902674 13163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:05.905148 13163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:05.905382 13163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:05.905663 13163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:05.906069 13163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:05.906147 13163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.906209 13163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:05.906252 13163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:05.913045 13163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40913
I20260924 17:41:05.913326 13322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40913 every 8 connection(s)
I20260924 17:41:05.913571 13163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:05.923789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13163
I20260924 17:41:05.924098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=42415
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:41:05 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:05.932717 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.914497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:53533)
0924 17:41:05.919747 (+ 5250us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:05.919808 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:05.920479 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:05.921348 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:05.921353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:05.921629 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:05.921915 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.921931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.923482 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.923483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.924846 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:05.924850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.925855 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.925856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:05.925954 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.926528 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:05.926536 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:05.928032 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:05.930530 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.930537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.930540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.930898 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.931240 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:05.931241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:05.931242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:05.931393 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:05.931978 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:05.931986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:05.932174 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 17:41:05.932293 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:41:05.932459 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5195,"thread_start_us":86,"threads_started":1}
I20260924 17:41:05.933714 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:05.914546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53533 (local address 127.0.98.62:45549)
0924 17:41:05.920720 (+ 6174us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:05.920730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:05.920785 (+ 55us) server_negotiation.cc:407] Connection header received
0924 17:41:05.920963 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:05.920969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:05.921024 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:05.921169 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:05.922055 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.922985 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.924952 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:05.925653 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:05.925751 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:05.928156 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:05.928162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:05.928162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:05.928176 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:05.930402 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.931007 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.931011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.931011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.931155 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:05.931477 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:05.931478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:05.931478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:05.931836 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:05.931955 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:05.933388 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0924 17:41:05.933523 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6111,"thread_start_us":62,"threads_started":1}
I20260924 17:41:05.935424 13323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:05.935563 13323 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:05.935750 13323 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:05.936633 13323 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:41:05.936717 13323 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:05.936785 13323 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:05.937145 13323 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:05.937218 13323 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:05.937567 13323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:41:06.038338 13327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:06.038679 13327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:06.038734 13327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:06.040256 13327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:06.040362 13327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:06.040424 13327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:06.042243 13327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42415
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:06.042675 13327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:06.042984 13327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:06.049074 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:06.049247 13335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:06.049561 13327 server_base.cc:1034] running on GCE node
W20260924 17:41:06.051198 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:06.051434 13327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:06.051731 13327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:06.052917 13327 hybrid_clock.cc:648] HybridClock initialized: now 1790271666052889 us; error 42 us; skew 500 ppm
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:06.060266 13327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:06.060923 13327 webserver.cc:460] Webserver started at http://127.0.98.2:42415/ using document root <none> and password file <none>
I20260924 17:41:06.061281 13327 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:06.061380 13327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:06.083490 13327 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:06.086961 13343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.087189 13327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 17:41:06.087301 13327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "62f859229f2f4ad4aa5e0239cc239049"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "288b7990c0af6411c9b58d6d2597e857"
server_key_iv: "9d1addc5f399b18f8478b6c3e1a1004d"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:06.087806 13327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:06.101336 13327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:06.103730 13327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:06.103917 13327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:06.104192 13327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:06.104571 13327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:06.104625 13327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.104669 13327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:06.104696 13327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.112208 13327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37979
I20260924 17:41:06.112262 13456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37979 every 8 connection(s)
I20260924 17:41:06.112857 13327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:06.122138 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:06.113666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45527 (local address 127.0.98.62:45549)
0924 17:41:06.113711 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:06.113713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:06.113959 (+ 246us) server_negotiation.cc:407] Connection header received
0924 17:41:06.114490 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:06.114491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:06.114508 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:06.114565 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:06.115194 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.115710 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.116748 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.116967 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.117053 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:06.119243 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:06.119245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:06.119245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:06.119254 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:06.120732 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.121150 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.121151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.121151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.121212 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.121469 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.121470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.121470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.121587 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:06.121665 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:06.121959 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 17:41:06.122045 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:41:06.122329 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:06.113668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:45527)
0924 17:41:06.113824 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:06.113869 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:06.114376 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:06.114645 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:06.114649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:06.114832 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:06.115086 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:06.115092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.115824 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.115826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:06.116665 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:06.116667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.117078 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.117079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:06.117205 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:06.117754 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:06.117763 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:06.119153 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:06.120843 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:06.120849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:06.120852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:06.121071 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:06.121280 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:06.121281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:06.121281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:06.121400 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:06.121655 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:06.121662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:06.121820 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 17:41:06.121939 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:41:06.122081 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260924 17:41:06.122628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13327
I20260924 17:41:06.122918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40049
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45549
--webserver_port=34513
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:06.122938 13457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:06.123024 13457 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.123243 13457 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:06.124001 13457 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:41:06.124090 13457 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.124212 13457 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:06.124917 13457 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.125008 13457 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:06.125366 13457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:41:06.246045 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:06.246454 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:06.246552 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:06.248426 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:06.248528 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:06.248590 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:06.250910 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34513
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:06.251384 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:06.251719 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:06.258312 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:06.258500 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:06.258783 13465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:06.259121 13460 server_base.cc:1034] running on GCE node
I20260924 17:41:06.259321 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:06.259627 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:06.260815 13460 hybrid_clock.cc:648] HybridClock initialized: now 1790271666260791 us; error 37 us; skew 500 ppm
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:06.264043 13460 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:06.264552 13460 webserver.cc:460] Webserver started at http://127.0.98.3:34513/ using document root <none> and password file <none>
I20260924 17:41:06.264827 13460 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:06.264891 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:06.287020 13460 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:06.288271 13477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.291144 13460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 17:41:06.291257 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "1d987cb3ae854e6b80952c8cbd0e86a6"
format_stamp: "Formatted at 2026-09-24 17:41:05 on dist-test-slave-d0pf"
server_key: "502136f3e2cb2a70f4442c89cd3869d2"
server_key_iv: "545a8f3c290010f58498ee548828b5ab"
server_key_version: "kuduclusterkey@0"
I20260924 17:41:06.291715 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:06.311913 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:06.314495 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:06.314738 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:06.315076 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:06.315496 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:06.315584 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.315652 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:06.315712 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:06.322204 13460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40049
I20260924 17:41:06.322633 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:06.327278 13590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40049 every 8 connection(s)
I20260924 17:41:06.329918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13460
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:06.338600 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:06.324204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41539 (local address 127.0.98.62:45549)
0924 17:41:06.324252 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:06.324255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:06.327699 (+ 3444us) server_negotiation.cc:407] Connection header received
0924 17:41:06.328338 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:06.328341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:06.328354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:06.328431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:06.329170 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.329944 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.331849 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.332157 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.332205 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:06.335264 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:06.335266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:06.335266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:06.335273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:06.336879 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.337376 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.337377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.337378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.337436 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.337731 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.337732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.337733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.337877 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:06.337936 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:06.338334 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 17:41:06.338501 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:41:06.339406 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:06.323452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:41539)
0924 17:41:06.327536 (+ 4084us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:06.327579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:06.328212 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:06.328517 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:06.328522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:06.328769 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:06.329056 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:06.329062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.330064 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.330065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:06.331729 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:06.331732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.332326 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.332327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:06.332475 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:06.333409 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:06.333421 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:06.335157 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:06.336973 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:06.336979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:06.336982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:06.337282 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:06.337504 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:06.337505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:06.337505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:06.337651 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:06.337971 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:06.337977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:06.338163 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 17:41:06.338466 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 17:41:06.339173 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4042,"thread_start_us":542,"threads_started":1}
I20260924 17:41:06.340071 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:06.330332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.98.62:45549)
0924 17:41:06.330615 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:06.330617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:06.330623 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:41:06.330643 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:06.330644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:06.330655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:06.330711 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:06.331506 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.332269 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.333689 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:06.334007 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:06.334065 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:06.336820 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:06.336821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:06.336821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:06.336827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:06.338557 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.338938 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.338939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.338940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.338996 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:06.339216 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:06.339217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:06.339217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:06.339335 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:06.339481 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:06.339881 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 17:41:06.339973 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":53,"threads_started":1}
I20260924 17:41:06.340230 13591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 17:41:06.340332 13591 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.340520 13591 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:06.341190 13591 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:41:06.341264 13591 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.341325 13591 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:06.341711 13591 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.341795 13591 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
W20260924 17:41:06.342092 13591 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 17:41:06 dist-test-slave-d0pf krb5kdc[12129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:06.938604 13323 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:06.938764 13323 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:06.939610 13125 ts_manager.cc:194] Registered new tserver with Master: 7c43254018dd41f0893a99317a6c7291 (127.0.98.1:40913)
I20260924 17:41:06.940634 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53533
I20260924 17:41:07.125578 13457 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:07.125775 13457 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:07.126215 13125 ts_manager.cc:194] Registered new tserver with Master: 62f859229f2f4ad4aa5e0239cc239049 (127.0.98.2:37979)
I20260924 17:41:07.126664 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45527
Sep 24 17:41:07 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 17:41:07.342283 13591 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:07.342458 13591 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 17:41:07.342830 13125 ts_manager.cc:194] Registered new tserver with Master: 1d987cb3ae854e6b80952c8cbd0e86a6 (127.0.98.3:40049)
I20260924 17:41:07.343423 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41539
I20260924 17:41:07.352710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:07.362967 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.353840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.0.98.62:45549)
0924 17:41:07.354281 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.354283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.354290 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:41:07.354315 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.354316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.354328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.355196 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:07.355238 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.356045 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.357132 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.357453 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.357825 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.359122 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:07.359124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:07.359125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:07.359133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:07.361323 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:07.361747 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:07.361748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:07.361749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:07.361828 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:07.362040 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:07.362041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:07.362042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:07.362183 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:07.362713 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.362755 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.362858 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":70,"threads_started":1}
I20260924 17:41:07.367604 13125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:41:07.369920 13125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:41:07.388114 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40049 (local address 127.0.0.1:58792)
0924 17:41:07.383249 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.383257 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:07.383287 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:07.383691 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:07.383694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:07.383696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:07.383778 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.383780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.385268 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.385269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.386563 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.386566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.387818 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.387819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.387909 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.387927 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 17:41:07.387973 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:41:07.388024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":65,"threads_started":1}
I20260924 17:41:07.388841 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.98.3:40049)
0924 17:41:07.383517 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.383522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.383532 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:07.383568 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.383569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.383585 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.383854 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:07.383925 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.385178 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.386665 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.387714 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.388605 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.388694 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.388723 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.388765 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":46,"threads_started":1}
I20260924 17:41:07.389737 13525 tablet_service.cc:1467] Processing CreateTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b (DEFAULT_TABLE table=test-table [id=d93b05e0837947ff9c28b9785d0af785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:07.390139 13525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e10cdd9c25a4f1492aea60f0848275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:07.393664 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37979 (local address 127.0.0.1:50326)
0924 17:41:07.383049 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.383091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:07.383154 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:07.389234 (+ 6080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:07.389235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:07.389237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:07.389283 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.389285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.390852 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.390853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.392007 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.392009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.393383 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.393384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.393466 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.393471 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:41:07.393508 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 17:41:07.393564 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":14,"thread_start_us":79,"threads_started":1}
I20260924 17:41:07.393571 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.0.1:40872)
0924 17:41:07.383380 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.383387 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:07.383418 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:07.387525 (+ 4107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:07.387526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:07.387528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:07.387574 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.387575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.390022 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.390023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.391193 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.391196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.393314 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.393315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.393396 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.393402 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:41:07.393438 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 17:41:07.393485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":51,"threads_started":1}
I20260924 17:41:07.393884 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.0.98.2:37979)
0924 17:41:07.389062 (+ 6763us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.389067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.389077 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:07.389111 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.389113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.389129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.389209 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:07.389383 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.390753 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.392108 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.392902 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.393612 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.393708 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.393740 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.393786 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6726,"thread_start_us":70,"threads_started":1}
I20260924 17:41:07.394477 13391 tablet_service.cc:1467] Processing CreateTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b (DEFAULT_TABLE table=test-table [id=d93b05e0837947ff9c28b9785d0af785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:07.394806 13391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e10cdd9c25a4f1492aea60f0848275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:07.395385 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.382709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.98.1:40913)
0924 17:41:07.387345 (+ 4636us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.387350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.387361 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:41:07.387398 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.387400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.387415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.387643 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:07.387723 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.388728 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.391299 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.392909 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.395142 (+ 2233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.395234 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.395262 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.395307 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4588,"thread_start_us":48,"threads_started":1}
I20260924 17:41:07.396620 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b (DEFAULT_TABLE table=test-table [id=d93b05e0837947ff9c28b9785d0af785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:07.396955 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e10cdd9c25a4f1492aea60f0848275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:07.398753 13626 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Bootstrap starting.
I20260924 17:41:07.399518 13626 tablet_bootstrap.cc:654] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:07.399812 13626 log.cc:824] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:07.400568 13626 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: No bootstrap required, opened a new log
I20260924 17:41:07.400635 13626 ts_tablet_manager.cc:1397] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:07.403264 13626 raft_consensus.cc:348] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.403446 13626 raft_consensus.cc:374] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:07.403478 13626 raft_consensus.cc:729] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f859229f2f4ad4aa5e0239cc239049, State: Initialized, Role: FOLLOWER
I20260924 17:41:07.403800 13626 consensus_queue.cc:260] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.404162 13626 ts_tablet_manager.cc:1428] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:41:07.404326 13457 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260924 17:41:07.405371 13628 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Bootstrap starting.
I20260924 17:41:07.406543 13628 tablet_bootstrap.cc:654] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:07.406935 13628 log.cc:824] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:07.407198 13625 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Bootstrap starting.
I20260924 17:41:07.407859 13628 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: No bootstrap required, opened a new log
I20260924 17:41:07.407881 13625 tablet_bootstrap.cc:654] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:07.407929 13628 ts_tablet_manager.cc:1397] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 17:41:07.408181 13625 log.cc:824] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:07.411013 13628 raft_consensus.cc:348] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.411692 13628 raft_consensus.cc:374] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:07.411778 13628 raft_consensus.cc:729] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c43254018dd41f0893a99317a6c7291, State: Initialized, Role: FOLLOWER
I20260924 17:41:07.411980 13628 consensus_queue.cc:260] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.412318 13628 ts_tablet_manager.cc:1428] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 17:41:07.412489 13323 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
W20260924 17:41:07.416744 13324 tablet.cc:2369] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:07.417637 13625 tablet_bootstrap.cc:492] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: No bootstrap required, opened a new log
I20260924 17:41:07.417719 13625 ts_tablet_manager.cc:1397] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260924 17:41:07.422197 13625 raft_consensus.cc:348] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.422470 13625 raft_consensus.cc:374] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:07.422549 13625 raft_consensus.cc:729] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d987cb3ae854e6b80952c8cbd0e86a6, State: Initialized, Role: FOLLOWER
I20260924 17:41:07.422749 13625 consensus_queue.cc:260] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.423142 13625 ts_tablet_manager.cc:1428] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 17:41:07.423309 13591 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
Sep 24 17:41:07 dist-test-slave-d0pf krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 17:41:07.578298 13592 tablet.cc:2369] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:41:07.614711 13458 tablet.cc:2369] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:07.716632 13632 raft_consensus.cc:482] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:07.716786 13632 raft_consensus.cc:504] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.717175 13632 leader_election.cc:288] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f859229f2f4ad4aa5e0239cc239049 (127.0.98.2:37979), 1d987cb3ae854e6b80952c8cbd0e86a6 (127.0.98.3:40049)
I20260924 17:41:07.722673 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.717987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51395 (local address 127.0.98.2:37979)
0924 17:41:07.718035 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.718037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.718989 (+ 952us) server_negotiation.cc:407] Connection header received
0924 17:41:07.719140 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.719142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.719155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.719186 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:07.719448 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.720167 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.721671 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.722248 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.722299 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.722325 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.722546 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.722593 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:41:07.723307 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.717490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37979 (local address 127.0.98.1:51395)
0924 17:41:07.718895 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.718909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:07.719023 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:07.719256 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:07.719257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:07.719258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:07.719368 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.719369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.720277 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.720278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.721553 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.721557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.722391 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.722392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.722465 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.722469 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:07.723127 (+ 658us) client_negotiation.cc:240] Negotiation successful
0924 17:41:07.723187 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":109,"threads_started":1}
I20260924 17:41:07.723949 13411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e10cdd9c25a4f1492aea60f0848275b" candidate_uuid: "7c43254018dd41f0893a99317a6c7291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f859229f2f4ad4aa5e0239cc239049" is_pre_election: true
I20260924 17:41:07.724129 13411 raft_consensus.cc:2454] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c43254018dd41f0893a99317a6c7291 in term 0.
I20260924 17:41:07.724505 13210 leader_election.cc:302] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f859229f2f4ad4aa5e0239cc239049, 7c43254018dd41f0893a99317a6c7291; no voters:
I20260924 17:41:07.724644 13632 raft_consensus.cc:2781] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:07.724686 13632 raft_consensus.cc:482] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:07.724709 13632 raft_consensus.cc:3037] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:07.725597 13632 raft_consensus.cc:504] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.725749 13632 leader_election.cc:288] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [CANDIDATE]: Term 1 election: Requested vote from peers 62f859229f2f4ad4aa5e0239cc239049 (127.0.98.2:37979), 1d987cb3ae854e6b80952c8cbd0e86a6 (127.0.98.3:40049)
I20260924 17:41:07.726119 13411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e10cdd9c25a4f1492aea60f0848275b" candidate_uuid: "7c43254018dd41f0893a99317a6c7291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f859229f2f4ad4aa5e0239cc239049"
I20260924 17:41:07.726203 13411 raft_consensus.cc:3037] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:07.726643 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.717827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59051 (local address 127.0.98.3:40049)
0924 17:41:07.717894 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.717896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.723202 (+ 5306us) server_negotiation.cc:407] Connection header received
0924 17:41:07.723309 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.723311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.723393 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.723458 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:07.723716 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.724352 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.725647 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.726242 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.726290 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.726300 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.726529 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.726576 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:41:07.726871 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.717722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40049 (local address 127.0.98.1:59051)
0924 17:41:07.723122 (+ 5400us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.723136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:07.723232 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:07.723563 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:07.723565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:07.723567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:07.723633 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.723635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.724444 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.724445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.725522 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:07.725525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.726373 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.726374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:07.726443 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.726446 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:07.726749 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 17:41:07.726792 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5379,"thread_start_us":71,"threads_started":1}
I20260924 17:41:07.727336 13542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e10cdd9c25a4f1492aea60f0848275b" candidate_uuid: "7c43254018dd41f0893a99317a6c7291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" is_pre_election: true
I20260924 17:41:07.727499 13542 raft_consensus.cc:2454] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c43254018dd41f0893a99317a6c7291 in term 0.
I20260924 17:41:07.727361 13541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e10cdd9c25a4f1492aea60f0848275b" candidate_uuid: "7c43254018dd41f0893a99317a6c7291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6"
I20260924 17:41:07.727572 13541 raft_consensus.cc:3037] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:07.728447 13541 raft_consensus.cc:2454] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c43254018dd41f0893a99317a6c7291 in term 1.
I20260924 17:41:07.728749 13211 leader_election.cc:302] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d987cb3ae854e6b80952c8cbd0e86a6, 7c43254018dd41f0893a99317a6c7291; no voters:
I20260924 17:41:07.729064 13632 raft_consensus.cc:2781] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:07.729231 13632 raft_consensus.cc:686] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [term 1 LEADER]: Becoming Leader. State: Replica: 7c43254018dd41f0893a99317a6c7291, State: Running, Role: LEADER
I20260924 17:41:07.729326 13632 consensus_queue.cc:237] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } }
I20260924 17:41:07.730237 13125 catalog_manager.cc:5495] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c43254018dd41f0893a99317a6c7291 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7c43254018dd41f0893a99317a6c7291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c43254018dd41f0893a99317a6c7291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20260924 17:41:07.730938 13411 raft_consensus.cc:2454] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c43254018dd41f0893a99317a6c7291 in term 1.
I20260924 17:41:07.739509 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.735701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.0.98.1:40913)
0924 17:41:07.735751 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:07.735753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:07.735937 (+ 184us) server_negotiation.cc:407] Connection header received
0924 17:41:07.736065 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:07.736066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:07.736091 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:07.736172 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:07.736387 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.737126 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.738264 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:07.738567 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:07.738638 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:07.738858 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:07.739000 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:07.739159 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:07.739400 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 17:41:07.739426 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:41:07.742007 13541 raft_consensus.cc:1261] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 1 FOLLOWER]: Refusing update from remote peer 7c43254018dd41f0893a99317a6c7291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:07.742451 13638 consensus_queue.cc:1035] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d987cb3ae854e6b80952c8cbd0e86a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:07.744853 13411 raft_consensus.cc:1261] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 1 FOLLOWER]: Refusing update from remote peer 7c43254018dd41f0893a99317a6c7291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:07.745230 13638 consensus_queue.cc:1035] T 4e10cdd9c25a4f1492aea60f0848275b P 7c43254018dd41f0893a99317a6c7291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f859229f2f4ad4aa5e0239cc239049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:07.749858 13641 mvcc.cc:205] Tried to move back new op lower bound from 7332952751067779072 to 7332952751019876352. Current Snapshot: MvccSnapshot[applied={T|T < 7332952751067779072}]
I20260924 17:41:07.750150 13644 mvcc.cc:205] Tried to move back new op lower bound from 7332952751067779072 to 7332952751019876352. Current Snapshot: MvccSnapshot[applied={T|T < 7332952751067779072}]
I20260924 17:41:07.752617 13642 mvcc.cc:205] Tried to move back new op lower bound from 7332952751067779072 to 7332952751019876352. Current Snapshot: MvccSnapshot[applied={T|T < 7332952751067779072}]
I20260924 17:41:07.755954 13125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:07.756074 13125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:07.756991 13125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 482335843e9e4f99b39b7567c7d2529c: Sending DeleteTablet for 3 replicas of tablet 4e10cdd9c25a4f1492aea60f0848275b
I20260924 17:41:07.757512 13391 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50326
I20260924 17:41:07.757786 13525 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58792
I20260924 17:41:07.758208 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e10cdd9c25a4f1492aea60f0848275b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40872
I20260924 17:41:07.758946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13163
I20260924 17:41:07.761983 13650 tablet_replica.cc:330] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: stopping tablet replica
I20260924 17:41:07.762163 13651 tablet_replica.cc:330] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: stopping tablet replica
I20260924 17:41:07.762427 13651 raft_consensus.cc:2229] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:07.762625 13650 raft_consensus.cc:2229] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:07.762782 13651 raft_consensus.cc:2258] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:07.762986 13650 raft_consensus.cc:2258] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:07.764010 13650 ts_tablet_manager.cc:1905] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:07.770167 13650 ts_tablet_manager.cc:1918] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:07.770351 13650 log.cc:1196] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/4e10cdd9c25a4f1492aea60f0848275b
I20260924 17:41:07.770524 13651 ts_tablet_manager.cc:1905] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:07.770744 13650 ts_tablet_manager.cc:1939] T 4e10cdd9c25a4f1492aea60f0848275b P 62f859229f2f4ad4aa5e0239cc239049: Deleting consensus metadata
I20260924 17:41:07.771327 13109 catalog_manager.cc:4841] TS 62f859229f2f4ad4aa5e0239cc239049 (127.0.98.2:37979): tablet 4e10cdd9c25a4f1492aea60f0848275b (table test-table [id=d93b05e0837947ff9c28b9785d0af785]) successfully deleted
W20260924 17:41:07.771808 13111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40913 (error 108)
I20260924 17:41:07.772444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13327
I20260924 17:41:07.773221 13651 ts_tablet_manager.cc:1918] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:07.773319 13651 log.cc:1196] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/4e10cdd9c25a4f1492aea60f0848275b
I20260924 17:41:07.773612 13651 ts_tablet_manager.cc:1939] T 4e10cdd9c25a4f1492aea60f0848275b P 1d987cb3ae854e6b80952c8cbd0e86a6: Deleting consensus metadata
I20260924 17:41:07.779246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13460
I20260924 17:41:07.787783 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:07.787630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.0.1:40884)
0924 17:41:07.787691 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:07.787732 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 17:41:07.788082 13111 catalog_manager.cc:4584] TS 7c43254018dd41f0893a99317a6c7291 (127.0.98.1:40913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e10cdd9c25a4f1492aea60f0848275b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40913: connect: Connection refused (error 111)
I20260924 17:41:07.788371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13092
I20260924 17:41:07.794744 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 17:41:08.139609 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 17:41:08.139704 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 17:41:08.484786 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 17:41:08.485 UTC [12159] LOG: received smart shutdown request
2026-09-24 17:41:08.488 UTC [12159] LOG: background worker "logical replication launcher" (PID 12168) exited with exit code 1
2026-09-24 17:41:08.489 UTC [12163] LOG: shutting down
2026-09-24 17:41:08.668 UTC [12159] LOG: database system is shut down
2026-09-24T17:41:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48151 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:41:08 dist-test-slave-d0pf krb5kdc[13662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:41:08 dist-test-slave-d0pf krb5kdc[13662](info): set up 2 sockets
Sep 24 17:41:08 dist-test-slave-d0pf krb5kdc[13662](info): commencing operation
krb5kdc: starting...
W20260924 17:41:09.831105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:41:09 dist-test-slave-d0pf krb5kdc[13662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:41:09.954437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:10.036544 13675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:10.036774 13675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:10.036799 13675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:10.037994 13675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:10.038040 13675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:10.038055 13675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:10.038069 13675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:10.038081 13675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:10.039691 13675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:10.039955 13675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:10.040187 13675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:10.045405 13680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:10.045439 13683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:10.045444 13681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:10.046028 13675 server_base.cc:1034] running on GCE node
I20260924 17:41:10.046299 13675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:10.046581 13675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:10.047741 13675 hybrid_clock.cc:648] HybridClock initialized: now 1790271670047730 us; error 34 us; skew 500 ppm
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:10.050391 13675 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:10.050889 13675 webserver.cc:460] Webserver started at http://127.0.98.62:33745/ using document root <none> and password file <none>
I20260924 17:41:10.051198 13675 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:10.051271 13675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:10.051412 13675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:10.052488 13675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "e2bfb3f11595d3deb6f9fee695fd86a1"
server_key_iv: "9f0fe5c99ff49ec85a1ff8058908a779"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.052856 13675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "e2bfb3f11595d3deb6f9fee695fd86a1"
server_key_iv: "9f0fe5c99ff49ec85a1ff8058908a779"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.054211 13675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:41:10.055015 13690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:10.055275 13675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:10.055382 13675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "e2bfb3f11595d3deb6f9fee695fd86a1"
server_key_iv: "9f0fe5c99ff49ec85a1ff8058908a779"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.055473 13675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:10.071005 13675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:10.073153 13675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:10.073334 13675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:10.077523 13675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44673
I20260924 17:41:10.077558 13742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44673 every 8 connection(s)
I20260924 17:41:10.077936 13675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:10.078814 13743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:10.079468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13675
I20260924 17:41:10.079571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:41:10.079762 392 external_mini_cluster.cc:1428] Setting key c89599db3fbff9f49cd3d4ccbfd7ac8b
I20260924 17:41:10.081245 13743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:10.094714 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.082147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.98.62:44673)
0924 17:41:10.082360 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.082365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.082470 (+ 105us) server_negotiation.cc:407] Connection header received
0924 17:41:10.083023 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.083026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.083072 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.083177 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.083767 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.084588 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.085522 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.085855 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.085970 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.090763 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.090776 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.090776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.090785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.092800 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.093364 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.093367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.093367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.093495 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.093828 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.093829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.093829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.094078 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.094154 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.094422 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.094536 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":121,"threads_started":1}
I20260924 17:41:10.094957 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.082088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44673 (local address 127.0.0.1:49330)
0924 17:41:10.082297 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.082347 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.082858 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.083198 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.083201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.083499 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.083659 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.083664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.084721 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.084722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.085420 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.085422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.085951 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.085952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.086018 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.086680 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.086689 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.090647 (+ 3958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.092915 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.092924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.092927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.093250 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.093592 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.093593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.093594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.093726 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.094162 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.094169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.094256 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.094378 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.094632 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":59,"thread_start_us":98,"threads_started":1}
I20260924 17:41:10.095764 13743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.096160 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.096024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45456)
0924 17:41:10.096080 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.096134 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 17:41:10.096403 13691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
I20260924 17:41:10.096921 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.096814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45470)
0924 17:41:10.096867 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.096897 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 17:41:10.097038 13743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 17:41:10.107331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41841
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:10.146309 13743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } attempt: 1
I20260924 17:41:10.146936 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.146789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45482)
0924 17:41:10.146861 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.146904 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 17:41:10.147344 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.147244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45492)
0924 17:41:10.147293 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.147317 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:41:10.147434 13743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
W20260924 17:41:10.190552 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:10.190790 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:10.190815 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:10.192128 13750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:10.192195 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:10.192211 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:10.192224 13750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:10.192237 13750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:10.193909 13750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:10.194443 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:10.194743 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:10.199878 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:10.199905 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:10.200059 13750 server_base.cc:1034] running on GCE node
W20260924 17:41:10.199905 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:10.200376 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:10.200743 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:10.201931 13750 hybrid_clock.cc:648] HybridClock initialized: now 1790271670201924 us; error 37 us; skew 500 ppm
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:10.204638 13750 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 17:41:10.205173 13750 webserver.cc:460] Webserver started at http://127.0.98.61:38419/ using document root <none> and password file <none>
I20260924 17:41:10.205427 13750 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:10.205497 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:10.205651 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:10.206736 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data/instance:
uuid: "02ba6db5f2874aa8a3ce7a836e6980e7"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "570e0950465a3f6c4c91bbca417bcf51"
server_key_iv: "0a08f7304edd76a900c8ec76d0bb9a26"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.207175 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal/instance:
uuid: "02ba6db5f2874aa8a3ce7a836e6980e7"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "570e0950465a3f6c4c91bbca417bcf51"
server_key_iv: "0a08f7304edd76a900c8ec76d0bb9a26"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.208591 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:10.209386 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:10.209586 13750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:10.209709 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "02ba6db5f2874aa8a3ce7a836e6980e7"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "570e0950465a3f6c4c91bbca417bcf51"
server_key_iv: "0a08f7304edd76a900c8ec76d0bb9a26"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.209795 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:10.215591 13743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } attempt: 2
I20260924 17:41:10.216045 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.215916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45494)
0924 17:41:10.215976 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.216016 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:41:10.216502 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.216388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45510)
0924 17:41:10.216440 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.216470 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 17:41:10.216596 13743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41841: connect: Connection refused (error 111)
I20260924 17:41:10.227578 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:10.229843 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:10.230034 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:10.234536 13750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41841
I20260924 17:41:10.234562 13817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41841 every 8 connection(s)
I20260924 17:41:10.234944 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
I20260924 17:41:10.235828 13818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:10.237442 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.242548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13750
I20260924 17:41:10.242667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-1/wal/instance
I20260924 17:41:10.242895 392 external_mini_cluster.cc:1428] Setting key 7d24237a6c70154666bb91e06b51e57b
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:10.248333 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.238226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.0.98.62:44673)
0924 17:41:10.238303 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.238305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.238626 (+ 321us) server_negotiation.cc:407] Connection header received
0924 17:41:10.239469 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.239471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.239487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.239574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.240243 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.240743 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.241929 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.242137 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.242204 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.244879 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.244881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.244881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.244891 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.246610 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.247188 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.247189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.247189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.247250 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.247577 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.247578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.247578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.247695 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.247756 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.248168 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.248264 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 17:41:10.248618 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.238255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44673 (local address 127.0.0.1:49332)
0924 17:41:10.238465 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.238518 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.239318 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.239646 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.239649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.239914 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.240112 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.240116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.240849 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.240850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.241827 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.241830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.242219 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.242220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.242398 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.243025 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.243085 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.244751 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.246717 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.246728 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.246731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.247091 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.247337 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.247338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.247339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.247497 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.247763 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.247770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.247969 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.248096 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.248314 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260924 17:41:10.249258 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:41:10.259280 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.249615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45520)
0924 17:41:10.249668 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.249679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.249724 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.250034 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.250036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.250188 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.250272 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.250274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.251294 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.251295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.251844 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.251846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.252391 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.252392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.252458 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.252980 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.252984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.255613 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.257764 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.257765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.257766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.258014 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.258381 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.258382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.258382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.258458 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.258862 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.258864 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.258921 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.259004 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.259135 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:41:10.259544 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.249709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.0.98.61:41841)
0924 17:41:10.249870 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.249875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.249897 (+ 22us) server_negotiation.cc:407] Connection header received
0924 17:41:10.249915 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.249917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.249927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.249983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.250381 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.251209 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.251937 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.252270 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.252342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.255715 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.255729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.255729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.255738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.257629 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.258191 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.258200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.258201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.258299 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.258531 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.258531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.258532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.258782 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.258856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.259140 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.259286 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":110,"threads_started":1}
I20260924 17:41:10.260393 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.260856 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.260693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44500)
0924 17:41:10.260760 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.260825 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 17:41:10.261039 13768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
I20260924 17:41:10.261667 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.261543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44514)
0924 17:41:10.261602 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.261636 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 17:41:10.261791 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 17:41:10.274410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34145
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:10.307775 13743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } attempt: 3
I20260924 17:41:10.311124 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } attempt: 1
I20260924 17:41:10.311664 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.311490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44518)
0924 17:41:10.311565 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.311616 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 17:41:10.312070 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.311948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44532)
0924 17:41:10.312010 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.312042 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 17:41:10.312278 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:41:10.315572 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.308265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.98.61:41841)
0924 17:41:10.308342 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.308345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.308358 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:41:10.308391 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.308392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.308405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.308480 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.308835 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.309467 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.310215 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.310438 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.310522 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.312904 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.312905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.312905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.312911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.314401 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.314802 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.314803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.314803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.314854 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.315092 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.315093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.315093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.315211 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.315285 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.315447 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.315514 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:41:10.315619 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.308169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45526)
0924 17:41:10.308236 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.308249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.308336 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.308486 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.308487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.308649 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.308714 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.308716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.309597 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.309599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.310121 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.310123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.310520 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.310522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.310605 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.311223 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.311229 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.312808 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.314496 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.314497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.314498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.314727 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.314918 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.314918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.314919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.314985 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.315294 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.315295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.315348 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.315421 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.315527 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:41:10.316020 13743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.316416 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.316304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44546)
0924 17:41:10.316355 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.316389 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:41:10.316753 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.316669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44550)
0924 17:41:10.316711 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.316736 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:41:10.316850 13743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
I20260924 17:41:10.348043 13743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } attempt: 1
I20260924 17:41:10.348560 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.348414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44560)
0924 17:41:10.348477 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.348523 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:41:10.348966 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.348859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44574)
0924 17:41:10.348908 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.348942 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:41:10.349069 13743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
W20260924 17:41:10.360430 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:10.360651 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:10.360674 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:10.361852 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:10.361896 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:10.361912 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:10.361924 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:10.361976 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:10.363737 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44673,127.0.98.61:41841,127.0.98.60:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:10.364004 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:10.364249 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:10.369151 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:10.369149 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:10.369212 13825 server_base.cc:1034] running on GCE node
W20260924 17:41:10.369151 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:10.369699 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:10.370021 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:10.371124 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790271670371103 us; error 70 us; skew 500 ppm
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:10.373693 13825 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 17:41:10.374110 13825 webserver.cc:460] Webserver started at http://127.0.98.60:45015/ using document root <none> and password file <none>
I20260924 17:41:10.374341 13825 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:10.374385 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:10.374475 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:10.375567 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data/instance:
uuid: "82911777493346158b9a734c980092ec"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "2f91f8d3e8af33c59619ac30bbe1280d"
server_key_iv: "7e9584afc1c6c009c0dcb429dd92f8ee"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.375878 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal/instance:
uuid: "82911777493346158b9a734c980092ec"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "2f91f8d3e8af33c59619ac30bbe1280d"
server_key_iv: "7e9584afc1c6c009c0dcb429dd92f8ee"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.377172 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:10.377955 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:10.378239 13825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:10.378321 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "82911777493346158b9a734c980092ec"
format_stamp: "Formatted at 2026-09-24 17:41:10 on dist-test-slave-d0pf"
server_key: "2f91f8d3e8af33c59619ac30bbe1280d"
server_key_iv: "7e9584afc1c6c009c0dcb429dd92f8ee"
server_key_version: "encryptionkey@0"
I20260924 17:41:10.378382 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:10.380435 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } attempt: 2
I20260924 17:41:10.380932 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.380776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44590)
0924 17:41:10.380862 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.380901 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 17:41:10.381368 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.381243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44602)
0924 17:41:10.381310 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.381346 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 17:41:10.381503 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34145: connect: Connection refused (error 111)
I20260924 17:41:10.388031 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:10.390317 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:10.390441 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:10.394667 13825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34145
I20260924 17:41:10.394757 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34145 every 8 connection(s)
I20260924 17:41:10.395166 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
I20260924 17:41:10.396117 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:10.397825 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.399530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13825
I20260924 17:41:10.399657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271516406493-392-0/minicluster-data/master-2/wal/instance
I20260924 17:41:10.399848 392 external_mini_cluster.cc:1428] Setting key 05bbd2f9c28519efbc33861a91cb0227
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:41:10.408547 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.398662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.0.98.62:44673)
0924 17:41:10.398735 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.398737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.398970 (+ 233us) server_negotiation.cc:407] Connection header received
0924 17:41:10.399530 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.399532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.399544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.399602 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.400512 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.401364 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.402791 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.403106 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.403190 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.405523 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.405525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.405525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.405533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.406978 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.407520 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.407521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.407521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.407574 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.407908 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.407909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.407909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.408027 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.408083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.408409 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.408479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:41:10.408916 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.398640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44673 (local address 127.0.0.1:49334)
0924 17:41:10.398846 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.398883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.399418 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.399684 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.399688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.400071 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.400408 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.400417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.401455 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.401456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.402619 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.402622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.403199 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.403200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.403337 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.403919 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.403930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.405424 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.407096 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.407103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.407105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.407438 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.407675 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.407676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.407676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.407827 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.408099 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.408104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.408259 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.408382 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.408577 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260924 17:41:10.409456 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.410089 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.400258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.98.60:34145)
0924 17:41:10.400467 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.400472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.400493 (+ 21us) server_negotiation.cc:407] Connection header received
0924 17:41:10.400541 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.400543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.400559 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.400610 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.401368 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.402828 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.403781 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.404252 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.404327 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.406758 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.406766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.406767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.406773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.408615 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.409028 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.409032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.409033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.409107 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.409313 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.409314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.409314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.409465 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.409651 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.409895 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.409973 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":177,"thread_start_us":132,"threads_started":1}
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:10.416586 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.409825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.0.98.61:41841)
0924 17:41:10.409870 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.409872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.409880 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:10.409907 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.409908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.409916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.409961 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.410293 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.410966 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.411668 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.411875 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.411959 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.413918 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.413919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.413920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.413927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.415416 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.415779 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.415781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.415781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.415837 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.416063 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.416064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.416064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.416182 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.416251 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.416410 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.416489 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:41:10.416586 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.409739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41841 (local address 127.0.0.1:45534)
0924 17:41:10.409790 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.409802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.409845 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.409973 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.409974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.410136 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.410212 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.410214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.411065 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.411066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.411565 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.411566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.411967 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.411968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.412046 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.412468 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.412474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.413826 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.415518 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.415519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.415520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.415689 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.415928 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.415929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.415930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.415988 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.416255 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.416256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.416301 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.416392 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.416490 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:41:10.417059 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:41:10.418118 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.410858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.0.98.62:44673)
0924 17:41:10.410899 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.410901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.410910 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:41:10.410931 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.410932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.410941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.410995 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.411466 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.411956 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.412637 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.412838 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.412925 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.415261 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.415263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.415264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.415272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.416781 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.417209 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.417211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.417211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.417273 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.417551 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.417553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.417553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.417675 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.417902 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.417944 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.418018 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:41:10.424194 13743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } attempt: 2
I20260924 17:41:10.424256 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.417372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44630)
0924 17:41:10.417413 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.417424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.417460 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.417582 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.417583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.417715 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.417743 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.417744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.418568 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.418568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.419262 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.419265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.419665 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.419666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.419722 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.420091 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.420095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.421392 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.423239 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.423240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.423241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.423407 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.423633 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.423634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.423635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.423696 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.423951 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.423952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.423990 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.424059 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.424159 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 17:41:10.424932 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.417423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.0.98.60:34145)
0924 17:41:10.417457 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.417458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.417466 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:10.417517 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.417518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.417526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.417579 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.417816 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.418492 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.419336 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.419597 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.419647 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.421509 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.421510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.421510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.421516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.423103 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.423495 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.423496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.423497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.423548 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.423777 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.423778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.423779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.423889 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.423960 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.424082 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.424821 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":660,"server-negotiator.queue_time_us":17}
I20260924 17:41:10.426126 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.418731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.98.61:41841)
0924 17:41:10.418765 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.418767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.418774 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:41:10.418825 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.418826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.418835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.418898 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.419414 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.419968 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.420692 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.420867 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.420921 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.423185 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.423186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.423186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.423194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.424768 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.425199 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.425200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.425200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.425264 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.425575 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.425576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.425576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.425693 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.425899 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.425935 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.426010 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:41:10.426882 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: Bootstrap starting.
I20260924 17:41:10.427717 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:10.428023 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:41:10.428040 13893 log.cc:824] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:10.428871 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: No bootstrap required, opened a new log
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:41:10.431139 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.431460 13893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:10.431560 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82911777493346158b9a734c980092ec, State: Initialized, Role: FOLLOWER
I20260924 17:41:10.431789 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.432251 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:10.432574 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.424554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44632)
0924 17:41:10.424611 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.424623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.424666 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.424916 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.424917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.425091 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.425140 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.425142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.426027 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.426029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.426804 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.426806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.427492 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.427493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.427562 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.427983 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.427987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.429390 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.431438 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.431439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.431439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.431666 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.431881 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.431882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.431882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.431945 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.432254 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.432255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.432306 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.432358 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.432477 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:41:10.433305 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.424637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.0.98.60:34145)
0924 17:41:10.424786 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.424788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.424796 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:10.424814 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.424815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.424824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.425212 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.425259 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.425927 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.427070 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.427391 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.427488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.429485 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.429486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.429486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.429492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.431327 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.431752 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.431752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.431753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.431809 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.432016 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.432016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.432017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.432173 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.433009 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.433076 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.433176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260924 17:41:10.434314 13918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:41:10.434418 13918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:41:10.435520 13825 master_runner.cc:186] Error getting master registration for 127.0.98.62:44673: OK, instance_id { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" instance_seqno: 1790271670076365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:41:10.437959 13825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:41:10.438019 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.429676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.0.98.61:41841)
0924 17:41:10.429709 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.429711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.429736 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:41:10.430081 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.430082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.430094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.430149 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.431169 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.431916 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.432933 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.433245 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.433284 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.435022 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.435023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.435024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.435077 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.436699 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.437123 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.437124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.437124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.437184 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.437399 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.437400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.437400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.437529 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.437614 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.437828 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.437923 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:41:10.438697 13743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: Bootstrap starting.
I20260924 17:41:10.439529 13743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:10.439872 13743 log.cc:824] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:10.440060 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.440157 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.441661 13743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: No bootstrap required, opened a new log
I20260924 17:41:10.443171 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.429919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.0.98.62:44673)
0924 17:41:10.429960 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.429961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.435131 (+ 5170us) server_negotiation.cc:407] Connection header received
0924 17:41:10.435231 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.435232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.435242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.435269 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.436429 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.437137 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.438095 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.438348 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.438393 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.439982 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.439984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.439984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.439992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.441653 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.442062 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.442063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.442064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.442121 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.442315 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.442316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.442316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.442449 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.442540 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.442943 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.443026 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:41:10.444049 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.429864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.0.98.60:34145)
0924 17:41:10.430002 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.430004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.430607 (+ 603us) server_negotiation.cc:407] Connection header received
0924 17:41:10.430706 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.430707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.430716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.430745 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.431452 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.432101 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.433016 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.433322 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.433360 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.441096 (+ 7736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.441098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.441098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.441106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.442746 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.443206 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.443207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.443207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.443270 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.443471 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.443472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.443472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.443595 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.443671 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.443876 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.443964 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 17:41:10.444664 13743 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.444816 13743 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:10.444902 13743 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36628a7d7bb4021b969e2c5bbfdd9fc, State: Initialized, Role: FOLLOWER
I20260924 17:41:10.445043 13743 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.445317 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.445392 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.445393 13743 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: configured and running, proceeding with master startup.
W20260924 17:41:10.447188 13932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:41:10.447292 13932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:41:10.448055 13675 master_runner.cc:186] Error getting master registration for 127.0.98.61:41841: OK, instance_id { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" instance_seqno: 1790271670233256 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:41:10.450852 13675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:41:10.456322 13921 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:10.456444 13921 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.456694 13921 leader_election.cc:288] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ba6db5f2874aa8a3ce7a836e6980e7 (127.0.98.61:41841), 82911777493346158b9a734c980092ec (127.0.98.60:34145)
I20260924 17:41:10.457036 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82911777493346158b9a734c980092ec" is_pre_election: true
I20260924 17:41:10.457198 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36628a7d7bb4021b969e2c5bbfdd9fc in term 0.
I20260924 17:41:10.457490 13693 leader_election.cc:302] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82911777493346158b9a734c980092ec, d36628a7d7bb4021b969e2c5bbfdd9fc; no voters:
I20260924 17:41:10.457643 13921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:10.457739 13921 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:10.457764 13921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:10.458784 13921 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.458922 13921 leader_election.cc:288] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [CANDIDATE]: Term 1 election: Requested vote from peers 02ba6db5f2874aa8a3ce7a836e6980e7 (127.0.98.61:41841), 82911777493346158b9a734c980092ec (127.0.98.60:34145)
I20260924 17:41:10.459307 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82911777493346158b9a734c980092ec"
I20260924 17:41:10.459407 13867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:10.460318 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36628a7d7bb4021b969e2c5bbfdd9fc in term 1.
I20260924 17:41:10.460491 13693 leader_election.cc:302] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82911777493346158b9a734c980092ec, d36628a7d7bb4021b969e2c5bbfdd9fc; no voters:
I20260924 17:41:10.460646 13921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:10.460803 13921 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [term 1 LEADER]: Becoming Leader. State: Replica: d36628a7d7bb4021b969e2c5bbfdd9fc, State: Running, Role: LEADER
I20260924 17:41:10.460889 13921 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.461330 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36628a7d7bb4021b969e2c5bbfdd9fc. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.461472 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:10.461809 13936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:10.462224 13936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:10.465137 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [term 1 FOLLOWER]: Refusing update from remote peer d36628a7d7bb4021b969e2c5bbfdd9fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:10.465606 13934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:10.467127 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36628a7d7bb4021b969e2c5bbfdd9fc. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.467227 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.472199 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.472298 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:10.472658 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } attempt: 3
I20260924 17:41:10.472744 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.472826 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.475541 13936 catalog_manager.cc:1347] Generated new cluster ID: 3f4bb68a52604c9aa393b27a8600e101
I20260924 17:41:10.475590 13936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:10.477059 13938 mvcc.cc:205] Tried to move back new op lower bound from 7332952762222456832 to 7332952762208165888. Current Snapshot: MvccSnapshot[applied={T|T < 7332952762222456832 or (T in {7332952762222456832})}]
Sep 24 17:41:10 dist-test-slave-d0pf krb5kdc[13662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:41:10.483292 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.472922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34145 (local address 127.0.0.1:44654)
0924 17:41:10.473004 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:10.473016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:10.473054 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:10.473328 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:10.473329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:10.473518 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:10.473566 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.473569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.475425 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.475426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.476221 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:10.476223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.476764 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.476765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:10.476833 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.477321 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:10.477327 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:10.480111 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:10.482033 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.482034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.482034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.482272 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.482524 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:10.482525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:10.482525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:10.482593 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:10.482916 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:10.482917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:10.482976 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 17:41:10.483078 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 17:41:10.483186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 17:41:10.483675 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:10.473127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.98.60:34145)
0924 17:41:10.473180 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:10.473182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:10.473189 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:41:10.473209 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:10.473210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:10.473220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:10.473265 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:10.474389 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.475339 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.476333 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:10.476639 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:10.476697 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:10.480217 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:10.480218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:10.480218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:10.480226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:10.481855 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.482375 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.482376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.482377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.482446 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:10.482665 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:10.482666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:10.482666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:10.482809 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:10.482868 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:10.483503 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 17:41:10.483584 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:41:10.485219 13818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: Bootstrap starting.
I20260924 17:41:10.485983 13818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:10.486290 13818 log.cc:824] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:10.497246 13818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: No bootstrap required, opened a new log
I20260924 17:41:10.500748 13818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.500939 13818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:10.500977 13818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ba6db5f2874aa8a3ce7a836e6980e7, State: Initialized, Role: FOLLOWER
I20260924 17:41:10.501145 13818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } }
I20260924 17:41:10.501770 13818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:10.501729 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" is_pre_election: true
I20260924 17:41:10.501988 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7"
I20260924 17:41:10.501754 13790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:10.503525 13790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [term 1 FOLLOWER]: Refusing update from remote peer d36628a7d7bb4021b969e2c5bbfdd9fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:10.503898 13934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:10.504073 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.504165 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:41:10.504676 13954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:41:10.504724 13954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:41:10.505473 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36628a7d7bb4021b969e2c5bbfdd9fc. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.505617 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.505687 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.505784 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:41:10.514655 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36628a7d7bb4021b969e2c5bbfdd9fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44673 } } peers { permanent_uuid: "02ba6db5f2874aa8a3ce7a836e6980e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41841 } } peers { permanent_uuid: "82911777493346158b9a734c980092ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34145 } } }
I20260924 17:41:10.514781 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:10.514721 13956 mvcc.cc:205] Tried to move back new op lower bound from 7332952762222456832 to 7332952762208165888. Current Snapshot: MvccSnapshot[applied={T|T < 7332952762222456832}]
I20260924 17:41:10.516324 13936 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:41:10.517092 13936 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:10.529551 13936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36628a7d7bb4021b969e2c5bbfdd9fc: Generated new TSK 0
I20260924 17:41:10.529850 13936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:41:11.434901 13918 catalog_manager.cc:1259] Loaded cluster ID: 3f4bb68a52604c9aa393b27a8600e101
I20260924 17:41:11.435001 13918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: loading cluster ID for follower catalog manager: success
I20260924 17:41:11.435845 13918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: acquiring CA information for follower catalog manager: success
I20260924 17:41:11.436097 13918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82911777493346158b9a734c980092ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:41:11.505183 13954 catalog_manager.cc:1259] Loaded cluster ID: 3f4bb68a52604c9aa393b27a8600e101
I20260924 17:41:11.505263 13954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: loading cluster ID for follower catalog manager: success
I20260924 17:41:11.506040 13954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: acquiring CA information for follower catalog manager: success
I20260924 17:41:11.506300 13954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02ba6db5f2874aa8a3ce7a836e6980e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:41:11.588752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13675
I20260924 17:41:11.594425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13750
I20260924 17:41:11.599342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13825
2026-09-24T17:41:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2818 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:41:11 dist-test-slave-d0pf krb5kdc[13968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:41:11 dist-test-slave-d0pf krb5kdc[13968](info): set up 2 sockets
Sep 24 17:41:11 dist-test-slave-d0pf krb5kdc[13968](info): commencing operation
krb5kdc: starting...
W20260924 17:41:12.647119 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:41:12 dist-test-slave-d0pf krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:41:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:41:12.778437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:12.862521 13981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:12.862761 13981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:12.862805 13981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:12.864359 13981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:12.864442 13981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:12.864465 13981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:12.864483 13981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:12.864506 13981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:12.866212 13981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:12.866495 13981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:12.866736 13981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:12.871721 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:12.871712 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:12.871712 13986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:12.871996 13981 server_base.cc:1034] running on GCE node
I20260924 17:41:12.872256 13981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:12.872561 13981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:12.873790 13981 hybrid_clock.cc:648] HybridClock initialized: now 1790271672873763 us; error 33 us; skew 500 ppm
Sep 24 17:41:12 dist-test-slave-d0pf krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:12.876997 13981 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:12.877456 13981 webserver.cc:460] Webserver started at http://127.0.98.62:46489/ using document root <none> and password file <none>
I20260924 17:41:12.877693 13981 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:12.877764 13981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:12.877900 13981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:12.879019 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "ee786edc0591403a8d7a75ca5d43c98b"
format_stamp: "Formatted at 2026-09-24 17:41:12 on dist-test-slave-d0pf"
server_key: "468c479c4bc6d4efa27137812844e802"
server_key_iv: "af188950244cec7c04ac7c45e52f522f"
server_key_version: "encryptionkey@0"
I20260924 17:41:12.879551 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee786edc0591403a8d7a75ca5d43c98b"
format_stamp: "Formatted at 2026-09-24 17:41:12 on dist-test-slave-d0pf"
server_key: "468c479c4bc6d4efa27137812844e802"
server_key_iv: "af188950244cec7c04ac7c45e52f522f"
server_key_version: "encryptionkey@0"
I20260924 17:41:12.880967 13981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:41:12.881803 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:12.882045 13981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:12.882171 13981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "ee786edc0591403a8d7a75ca5d43c98b"
format_stamp: "Formatted at 2026-09-24 17:41:12 on dist-test-slave-d0pf"
server_key: "468c479c4bc6d4efa27137812844e802"
server_key_iv: "af188950244cec7c04ac7c45e52f522f"
server_key_version: "encryptionkey@0"
I20260924 17:41:12.882251 13981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:12.894445 13981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:12.896920 13981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:12.897040 13981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:12.900899 13981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40803
I20260924 17:41:12.900966 14048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40803 every 8 connection(s)
I20260924 17:41:12.901320 13981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:12.902280 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:12.903401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 13981
I20260924 17:41:12.903484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:41:12.903690 392 external_mini_cluster.cc:1428] Setting key 6ca66db661ecfec5885b1dab026ec228
I20260924 17:41:12.905421 14049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: Bootstrap starting.
I20260924 17:41:12.906044 14049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:12.906363 14049 log.cc:824] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:12.907140 14049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: No bootstrap required, opened a new log
I20260924 17:41:12.909437 14049 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } }
I20260924 17:41:12.909600 14049 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:12.909622 14049 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee786edc0591403a8d7a75ca5d43c98b, State: Initialized, Role: FOLLOWER
I20260924 17:41:12.909752 14049 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } }
I20260924 17:41:12.909823 14049 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:12.909845 14049 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:12.909909 14049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:12.910449 14049 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } }
I20260924 17:41:12.910555 14049 leader_election.cc:302] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee786edc0591403a8d7a75ca5d43c98b; no voters:
I20260924 17:41:12.910727 14049 leader_election.cc:288] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:41:12.910794 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:12.910974 14054 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [term 1 LEADER]: Becoming Leader. State: Replica: ee786edc0591403a8d7a75ca5d43c98b, State: Running, Role: LEADER
Sep 24 17:41:12 dist-test-slave-d0pf krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:12.910986 14049 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:12.911115 14054 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } }
I20260924 17:41:12.911659 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee786edc0591403a8d7a75ca5d43c98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } } }
I20260924 17:41:12.912264 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee786edc0591403a8d7a75ca5d43c98b. Latest consensus state: current_term: 1 leader_uuid: "ee786edc0591403a8d7a75ca5d43c98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee786edc0591403a8d7a75ca5d43c98b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40803 } } }
I20260924 17:41:12.911744 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:12.912838 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b [sys.catalog]: This master's current role is: LEADER
W20260924 17:41:12.913065 14069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:41:12.913239 14069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:41:12.913992 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:12.914217 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:12.915575 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:12.904239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.0.98.62:40803)
0924 17:41:12.904439 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:12.904449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:12.904497 (+ 48us) server_negotiation.cc:407] Connection header received
0924 17:41:12.905131 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:12.905141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:12.905205 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:12.905359 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:12.905985 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:12.907375 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:12.908109 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:12.908769 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:12.908901 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:12.911700 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:12.911703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:12.911704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:12.911715 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:12.913827 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:12.914236 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:12.914240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:12.914241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:12.914344 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:12.914576 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:12.914577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:12.914578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:12.914880 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:12.915095 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:12.915213 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:41:12.915352 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260924 17:41:12.915831 14071 catalog_manager.cc:1347] Generated new cluster ID: 771118fb056c4651be4eb1e9f55a4da9
I20260924 17:41:12.915867 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:12.916508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:41:12.931100 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:41:12.931929 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:12.936059 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee786edc0591403a8d7a75ca5d43c98b: Generated new TSK 0
I20260924 17:41:12.936264 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:41:12.970561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 13981
2026-09-24T17:41:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1369 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:41:13 dist-test-slave-d0pf krb5kdc[14076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:41:13 dist-test-slave-d0pf krb5kdc[14076](info): set up 2 sockets
Sep 24 17:41:13 dist-test-slave-d0pf krb5kdc[14076](info): commencing operation
krb5kdc: starting...
W20260924 17:41:14.011201 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:41:14.135083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:14.218556 14089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:14.218791 14089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:14.218815 14089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:14.219986 14089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:14.220027 14089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:14.220042 14089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:14.220054 14089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:14.220067 14089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:14.221817 14089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:14.222049 14089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:14.222227 14089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:14.227244 14095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.227238 14094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.227238 14097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:14.227538 14089 server_base.cc:1034] running on GCE node
I20260924 17:41:14.227890 14089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:14.228216 14089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:14.229372 14089 hybrid_clock.cc:648] HybridClock initialized: now 1790271674229362 us; error 32 us; skew 500 ppm
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:14.238240 14089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:14.238848 14089 webserver.cc:460] Webserver started at http://127.0.98.62:37567/ using document root <none> and password file <none>
I20260924 17:41:14.239142 14089 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:14.239215 14089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:14.239372 14089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:14.240507 14089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "0097c6a2676d4ac3828cce57a61de4b0"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "f0969d191c0f063a3b766af683d6554d"
server_key_iv: "c7f6e0dc6f6e3a9e81fe2fdaa9e4aab1"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.240985 14089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "0097c6a2676d4ac3828cce57a61de4b0"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "f0969d191c0f063a3b766af683d6554d"
server_key_iv: "c7f6e0dc6f6e3a9e81fe2fdaa9e4aab1"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.242492 14089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:41:14.243348 14104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.243520 14089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:14.243624 14089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0097c6a2676d4ac3828cce57a61de4b0"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "f0969d191c0f063a3b766af683d6554d"
server_key_iv: "c7f6e0dc6f6e3a9e81fe2fdaa9e4aab1"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.243698 14089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:14.250509 14089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:14.252485 14089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:14.252646 14089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:14.256783 14089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42383
I20260924 17:41:14.256825 14156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42383 every 8 connection(s)
I20260924 17:41:14.257186 14089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:14.258163 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:14.260142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14089
I20260924 17:41:14.260259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:41:14.260458 392 external_mini_cluster.cc:1428] Setting key dabcb73336252c10115c40dca9fc7f67
I20260924 17:41:14.260980 14157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap starting.
I20260924 17:41:14.261586 14157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:14.261902 14157 log.cc:824] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:14.262693 14157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: No bootstrap required, opened a new log
I20260924 17:41:14.264873 14157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:14.265033 14157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:14.265087 14157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Initialized, Role: FOLLOWER
I20260924 17:41:14.265232 14157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:14.265311 14157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:14.265353 14157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:14.265400 14157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:14.266093 14157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:14.266208 14157 leader_election.cc:302] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0097c6a2676d4ac3828cce57a61de4b0; no voters:
I20260924 17:41:14.266404 14157 leader_election.cc:288] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:41:14.266427 14162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:14.266541 14162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Running, Role: LEADER
I20260924 17:41:14.266693 14157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:14.266678 14162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:14.267145 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:41:14.267409 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:14.267331 14163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0097c6a2676d4ac3828cce57a61de4b0. Latest consensus state: current_term: 1 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:14.267853 14171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:14.267865 14163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:14.268438 14171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:14.270239 14171 catalog_manager.cc:1347] Generated new cluster ID: c17d876bcc93495db5709fd129f8e536
I20260924 17:41:14.270300 14171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:14.272048 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.261006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.98.62:42383)
0924 17:41:14.261187 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.261196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.261229 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:41:14.261654 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.261660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.261749 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.261893 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:14.262652 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.263650 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.264787 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.265285 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.265404 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.268365 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:14.268369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:14.268369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:14.268381 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:14.270465 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.270833 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.270836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.270837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.270930 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.271160 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.271161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.271161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.271405 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:14.271570 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.271680 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.271818 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260924 17:41:14.285965 14171 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:41:14.286737 14171 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:14.301522 14171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Generated new TSK 0
I20260924 17:41:14.301765 14171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:41:14.337779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:14.420431 14183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:14.420691 14183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:14.420732 14183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:14.421988 14183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:14.422060 14183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:14.422107 14183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:14.423708 14183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:14.424009 14183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:14.424265 14183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:14.429380 14191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.429410 14188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.429450 14189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:14.429572 14183 server_base.cc:1034] running on GCE node
I20260924 17:41:14.429801 14183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:14.430009 14183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:14.431160 14183 hybrid_clock.cc:648] HybridClock initialized: now 1790271674431147 us; error 28 us; skew 500 ppm
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:14.433792 14183 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:14.434228 14183 webserver.cc:460] Webserver started at http://127.0.98.1:40591/ using document root <none> and password file <none>
I20260924 17:41:14.434473 14183 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:14.434518 14183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:14.434671 14183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:14.435770 14183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "9723b24fbd53484098db4aea9a5a6065"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "72ed21d1981f4dc583e8cffa30592f24"
server_key_iv: "c66e48111e25c0a780ef167d973165b9"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.436115 14183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9723b24fbd53484098db4aea9a5a6065"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "72ed21d1981f4dc583e8cffa30592f24"
server_key_iv: "c66e48111e25c0a780ef167d973165b9"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.437451 14183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:14.438122 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.438308 14183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:41:14.438377 14183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "9723b24fbd53484098db4aea9a5a6065"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "72ed21d1981f4dc583e8cffa30592f24"
server_key_iv: "c66e48111e25c0a780ef167d973165b9"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.438462 14183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:14.478153 14183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:14.480216 14183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:14.480371 14183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:14.480623 14183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:14.480937 14183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:14.480970 14183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.481009 14183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:14.481037 14183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.486568 14183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34385
I20260924 17:41:14.486646 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34385 every 8 connection(s)
I20260924 17:41:14.486953 14183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:14.493686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14183
I20260924 17:41:14.493806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:41:14.494354 392 external_mini_cluster.cc:1428] Setting key 58c70bfbb23567efa9c2e5d01a73050e
I20260924 17:41:14.496894 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.487779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50515 (local address 127.0.98.62:42383)
0924 17:41:14.487836 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.487838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.488015 (+ 177us) server_negotiation.cc:407] Connection header received
0924 17:41:14.488509 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.488510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.488519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.488587 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:14.489520 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.490048 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.491114 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.491319 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.491382 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.493553 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:14.493555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:14.493556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:14.493565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:14.495341 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.495750 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.495751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.495752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.495818 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.496126 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.496127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.496127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.496283 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:14.496360 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.496660 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.496770 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:41:14.497169 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.487726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:50515)
0924 17:41:14.487889 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.487931 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.488426 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.488643 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.488647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.489211 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:14.489417 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.489422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.490136 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.490137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.490994 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.490996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.491400 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.491400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.491515 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.491997 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:14.492005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:14.493453 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:14.495432 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.495439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.495441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.495677 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.495920 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.495921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.495921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.496054 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.496349 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:14.496354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:14.496484 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.496625 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.496856 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260924 17:41:14.497650 14312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:14.497766 14312 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:14.497968 14312 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:14.498497 14121 ts_manager.cc:194] Registered new tserver with Master: 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:14.499250 14121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50515
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:41:14.519613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:14.601183 14317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:14.601445 14317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:14.601486 14317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:14.602690 14317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:14.602769 14317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:14.602851 14317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:14.604491 14317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:14.604775 14317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:14.605026 14317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:14.610313 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.610373 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.610556 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:14.610584 14317 server_base.cc:1034] running on GCE node
I20260924 17:41:14.610819 14317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:14.611135 14317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:14.612299 14317 hybrid_clock.cc:648] HybridClock initialized: now 1790271674612265 us; error 49 us; skew 500 ppm
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:14.614984 14317 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:14.615545 14317 webserver.cc:460] Webserver started at http://127.0.98.2:45665/ using document root <none> and password file <none>
I20260924 17:41:14.615792 14317 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:14.615841 14317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:14.615967 14317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:14.617065 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "827d464350014249b12b51714b25444a"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "89235eabd19761481f32320b650b12c3"
server_key_iv: "a4e70a46a8c98bba1c7bc5886fcf56a4"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.617440 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "827d464350014249b12b51714b25444a"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "89235eabd19761481f32320b650b12c3"
server_key_iv: "a4e70a46a8c98bba1c7bc5886fcf56a4"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.618814 14317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 17:41:14.619701 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.619926 14317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:14.620000 14317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "827d464350014249b12b51714b25444a"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "89235eabd19761481f32320b650b12c3"
server_key_iv: "a4e70a46a8c98bba1c7bc5886fcf56a4"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.620146 14317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:14.666028 14317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:14.668435 14317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:14.668596 14317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:14.668838 14317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:14.669149 14317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:14.669199 14317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.669239 14317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:14.669279 14317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.675068 14317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40539
I20260924 17:41:14.675137 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40539 every 8 connection(s)
I20260924 17:41:14.675498 14317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:14.685325 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.676342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45407 (local address 127.0.98.62:42383)
0924 17:41:14.676430 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.676432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.676664 (+ 232us) server_negotiation.cc:407] Connection header received
0924 17:41:14.677283 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.677285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.677300 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.677372 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:14.678375 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.678873 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.679933 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.680129 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.680215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.682379 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:14.682381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:14.682382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:14.682391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:14.683851 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.684311 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.684312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.684313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.684373 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.684621 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.684622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.684623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.684749 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:14.684819 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.685103 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.685229 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 17:41:14.685428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14317
I20260924 17:41:14.685491 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.676307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:45407)
0924 17:41:14.676526 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.676564 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.677137 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.677455 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.677459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.678052 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:14.678253 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.678259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.678976 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.678978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.679826 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.679828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.680225 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.680227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.680352 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.680899 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:14.680907 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:14.682278 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:14.683954 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.683962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.683965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.684204 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.684441 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.684442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.684443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.684553 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.684821 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:14.684828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:14.684973 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.685080 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.685250 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":151,"threads_started":1}
I20260924 17:41:14.685539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:41:14.685734 392 external_mini_cluster.cc:1428] Setting key a3097481fbbd4b62351818214f2138e9
I20260924 17:41:14.685949 14446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:14.686061 14446 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:14.686259 14446 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:14.686637 14121 ts_manager.cc:194] Registered new tserver with Master: 827d464350014249b12b51714b25444a (127.0.98.2:40539)
I20260924 17:41:14.687110 14121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45407
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:41:14.709594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:14.800179 14451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:14.800489 14451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:14.800534 14451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:14.802243 14451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:14.802314 14451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:14.802368 14451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:14.804507 14451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:14.804819 14451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:14.805114 14451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:14.811380 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.811384 14457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:14.811375 14459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:14.812322 14451 server_base.cc:1034] running on GCE node
I20260924 17:41:14.812573 14451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:14.812876 14451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:14.814064 14451 hybrid_clock.cc:648] HybridClock initialized: now 1790271674814042 us; error 34 us; skew 500 ppm
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:14.817723 14451 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:14.818225 14451 webserver.cc:460] Webserver started at http://127.0.98.3:41397/ using document root <none> and password file <none>
I20260924 17:41:14.818467 14451 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:14.818512 14451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:14.818650 14451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:14.819834 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "274079640ce544d8a9b865326fd90710"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "5ed81c843bf505a94b141c41e86e521b"
server_key_iv: "257a44205aaf7bda8676df3ab6940d33"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.820204 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "274079640ce544d8a9b865326fd90710"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "5ed81c843bf505a94b141c41e86e521b"
server_key_iv: "257a44205aaf7bda8676df3ab6940d33"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.821534 14451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:14.822301 14466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.822474 14451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:41:14.822571 14451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "274079640ce544d8a9b865326fd90710"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "5ed81c843bf505a94b141c41e86e521b"
server_key_iv: "257a44205aaf7bda8676df3ab6940d33"
server_key_version: "encryptionkey@0"
I20260924 17:41:14.822641 14451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:14.845326 14451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:14.847931 14451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:14.848097 14451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:14.848351 14451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:14.848680 14451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:14.848721 14451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.848773 14451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:14.848800 14451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:14.854445 14451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46203
I20260924 17:41:14.854519 14579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46203 every 8 connection(s)
I20260924 17:41:14.854851 14451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:14.855459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14451
I20260924 17:41:14.855551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:41:14.855739 392 external_mini_cluster.cc:1428] Setting key 74f236ae11df2f83613e366bc2447831
Sep 24 17:41:14 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:14.865149 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.855657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52321 (local address 127.0.98.62:42383)
0924 17:41:14.855719 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.855721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.856089 (+ 368us) server_negotiation.cc:407] Connection header received
0924 17:41:14.856803 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.856804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.856816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.856877 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:14.858008 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.858526 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.859689 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.859910 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.859976 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.862241 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:14.862242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:14.862243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:14.862251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:14.863712 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.864137 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.864138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.864138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.864197 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.864506 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.864507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.864507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.864635 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:14.864707 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.864999 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.865076 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:41:14.865314 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.855651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:52321)
0924 17:41:14.855914 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.855977 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.856634 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.857021 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.857025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.857631 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:14.857883 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.857890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.858633 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.858634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.859601 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.859605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.860008 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.860009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.860138 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.860673 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:14.860682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:14.862134 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:14.863789 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.863796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.863798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.864053 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.864282 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:14.864283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:14.864284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:14.864416 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:14.864709 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:14.864714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:14.864845 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.864942 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.865082 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260924 17:41:14.865842 14580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:14.865965 14580 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:14.866153 14580 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:14.866549 14121 ts_manager.cc:194] Registered new tserver with Master: 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
I20260924 17:41:14.866941 14121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52321
I20260924 17:41:14.867775 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:14.875329 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.869064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.98.62:42383)
0924 17:41:14.869131 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.869133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.869203 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:41:14.869256 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.869257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.869266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.869314 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:14.869959 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.870442 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.871259 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.871448 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.871513 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.872659 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:14.872660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:14.872661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:14.872667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:14.874110 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.874468 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.874469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.874469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.874518 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:14.874758 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:14.874759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:14.874759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:14.874924 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:14.874999 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.875175 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.875239 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:41:14.877035 14121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:41:14.878700 14121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:41:14.885754 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.0.98.2:40539)
0924 17:41:14.881484 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.881489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.881682 (+ 193us) server_negotiation.cc:407] Connection header received
0924 17:41:14.881849 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.881852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.881868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.881944 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:14.882170 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.882974 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.884557 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.885356 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.885439 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.885504 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.885639 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.885680 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260924 17:41:14.885761 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.0.1:44290)
0924 17:41:14.881622 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.881631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.881708 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.882027 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.882027 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.882029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:14.882085 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.882086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.883149 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.883150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.884451 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.884454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.885435 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.885436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.885524 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.885540 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.885616 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.885672 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260924 17:41:14.886204 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.0.1:60202)
0924 17:41:14.881682 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.881692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.881752 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.882102 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.882103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.882105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:14.882139 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.882141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.883421 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.883422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.884487 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.884489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.885971 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.885972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.886034 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.886039 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.886088 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.886139 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":75,"threads_started":1}
I20260924 17:41:14.886482 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.0.1:59640)
0924 17:41:14.881467 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:14.881515 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:14.881591 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:14.881961 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:14.881964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:14.881967 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:14.882042 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.882044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.884574 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.884574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.885242 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:14.885244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.886254 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.886255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:14.886337 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.886341 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:14.886391 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:41:14.886426 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":81,"threads_started":1}
I20260924 17:41:14.886524 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.0.98.1:34385)
0924 17:41:14.881662 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.881668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.881738 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:41:14.881848 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.881850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.881867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.882029 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:14.882203 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.883320 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.884579 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.885357 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.885439 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.885504 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.886387 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.886438 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260924 17:41:14.886575 14380 tablet_service.cc:1467] Processing CreateTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 (DEFAULT_TABLE table=test-table [id=b03089f4fad945bdaea56f45fba8922f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:14.886946 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:14.881278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.98.3:46203)
0924 17:41:14.881563 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:14.881567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:14.881577 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:14.881733 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:14.881736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:14.881755 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:14.881838 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:14.882126 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.883309 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.885335 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:14.886087 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:14.886149 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:14.886219 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:14.886815 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 17:41:14.886863 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":91,"threads_started":1}
I20260924 17:41:14.886961 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee87946ce9d441f8e92c2b6c49b0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:14.887171 14246 tablet_service.cc:1467] Processing CreateTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 (DEFAULT_TABLE table=test-table [id=b03089f4fad945bdaea56f45fba8922f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:14.887462 14246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee87946ce9d441f8e92c2b6c49b0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:14.889642 14597 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Bootstrap starting.
I20260924 17:41:14.890024 14598 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Bootstrap starting.
I20260924 17:41:14.890352 14597 tablet_bootstrap.cc:654] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:14.890456 14514 tablet_service.cc:1467] Processing CreateTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 (DEFAULT_TABLE table=test-table [id=b03089f4fad945bdaea56f45fba8922f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:14.890666 14598 tablet_bootstrap.cc:654] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:14.890839 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee87946ce9d441f8e92c2b6c49b0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:14.890926 14597 log.cc:824] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:14.891278 14598 log.cc:824] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:14.891978 14598 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: No bootstrap required, opened a new log
I20260924 17:41:14.892024 14597 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: No bootstrap required, opened a new log
I20260924 17:41:14.892059 14598 ts_tablet_manager.cc:1397] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:14.892093 14597 ts_tablet_manager.cc:1397] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 17:41:14.893447 14598 raft_consensus.cc:348] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.893628 14598 raft_consensus.cc:374] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:14.893674 14598 raft_consensus.cc:729] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9723b24fbd53484098db4aea9a5a6065, State: Initialized, Role: FOLLOWER
I20260924 17:41:14.893841 14598 consensus_queue.cc:260] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.894150 14598 ts_tablet_manager.cc:1428] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:14.894285 14312 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
I20260924 17:41:14.894225 14597 raft_consensus.cc:348] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.894393 14597 raft_consensus.cc:374] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:14.894433 14597 raft_consensus.cc:729] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827d464350014249b12b51714b25444a, State: Initialized, Role: FOLLOWER
I20260924 17:41:14.894565 14597 consensus_queue.cc:260] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.894838 14597 ts_tablet_manager.cc:1428] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:41:14.894902 14446 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
I20260924 17:41:14.895150 14602 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Bootstrap starting.
I20260924 17:41:14.895923 14602 tablet_bootstrap.cc:654] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:14.896378 14602 log.cc:824] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:14.897094 14602 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: No bootstrap required, opened a new log
I20260924 17:41:14.897150 14602 ts_tablet_manager.cc:1397] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:14.898567 14602 raft_consensus.cc:348] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.898731 14602 raft_consensus.cc:374] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:14.898778 14602 raft_consensus.cc:729] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274079640ce544d8a9b865326fd90710, State: Initialized, Role: FOLLOWER
I20260924 17:41:14.898897 14602 consensus_queue.cc:260] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:14.899219 14602 ts_tablet_manager.cc:1428] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:41:14.899240 14580 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
W20260924 17:41:14.926203 14447 tablet.cc:2369] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:41:14.987712 14313 tablet.cc:2369] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:15.020581 14605 raft_consensus.cc:482] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:15.020699 14605 raft_consensus.cc:504] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.021113 14605 leader_election.cc:288] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:15.024439 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.021313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.98.3:56159)
0924 17:41:15.021363 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.021376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.021442 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.021607 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.021609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.021611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.021686 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.021688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.022443 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.022444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.023511 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.023513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.024154 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.024156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.024243 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.024246 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.024313 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.024360 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 17:41:15.024462 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.021402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56159 (local address 127.0.98.2:40539)
0924 17:41:15.021470 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.021472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.021483 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:41:15.021513 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.021514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.021528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.021589 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.021770 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.022370 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.023614 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.024042 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.024153 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.024166 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.024334 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.024380 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:41:15.024611 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.021387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.98.3:56731)
0924 17:41:15.021557 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.021566 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.021614 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.021802 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.021803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.021805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.021875 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.021876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.022765 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.022766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.023644 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.023646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.024339 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.024340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.024410 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.024414 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.024474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.024516 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260924 17:41:15.024616 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.021516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56731 (local address 127.0.98.1:34385)
0924 17:41:15.021561 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.021564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.021614 (+ 50us) server_negotiation.cc:407] Connection header received
0924 17:41:15.021667 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.021668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.021680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.021796 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.021964 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.022663 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.023720 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.024246 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.024322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.024331 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.024488 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.024531 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:41:15.024860 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee87946ce9d441f8e92c2b6c49b0cc7" candidate_uuid: "274079640ce544d8a9b865326fd90710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827d464350014249b12b51714b25444a" is_pre_election: true
I20260924 17:41:15.024930 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee87946ce9d441f8e92c2b6c49b0cc7" candidate_uuid: "274079640ce544d8a9b865326fd90710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9723b24fbd53484098db4aea9a5a6065" is_pre_election: true
I20260924 17:41:15.025034 14400 raft_consensus.cc:2454] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274079640ce544d8a9b865326fd90710 in term 0.
I20260924 17:41:15.025074 14266 raft_consensus.cc:2454] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274079640ce544d8a9b865326fd90710 in term 0.
I20260924 17:41:15.025297 14469 leader_election.cc:302] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a, 9723b24fbd53484098db4aea9a5a6065; no voters:
I20260924 17:41:15.025439 14605 raft_consensus.cc:2781] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:15.025491 14605 raft_consensus.cc:482] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:15.025517 14605 raft_consensus.cc:3037] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.026340 14605 raft_consensus.cc:504] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.026499 14605 leader_election.cc:288] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [CANDIDATE]: Term 1 election: Requested vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:15.026669 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee87946ce9d441f8e92c2b6c49b0cc7" candidate_uuid: "274079640ce544d8a9b865326fd90710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9723b24fbd53484098db4aea9a5a6065"
I20260924 17:41:15.026742 14266 raft_consensus.cc:3037] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.027467 14266 raft_consensus.cc:2454] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274079640ce544d8a9b865326fd90710 in term 1.
I20260924 17:41:15.027598 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee87946ce9d441f8e92c2b6c49b0cc7" candidate_uuid: "274079640ce544d8a9b865326fd90710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827d464350014249b12b51714b25444a"
I20260924 17:41:15.027659 14469 leader_election.cc:302] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274079640ce544d8a9b865326fd90710, 9723b24fbd53484098db4aea9a5a6065; no voters:
I20260924 17:41:15.027691 14400 raft_consensus.cc:3037] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.027751 14605 raft_consensus.cc:2781] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:15.027977 14605 raft_consensus.cc:686] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 1 LEADER]: Becoming Leader. State: Replica: 274079640ce544d8a9b865326fd90710, State: Running, Role: LEADER
I20260924 17:41:15.028115 14605 consensus_queue.cc:237] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.028724 14400 raft_consensus.cc:2454] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274079640ce544d8a9b865326fd90710 in term 1.
I20260924 17:41:15.029059 14121 catalog_manager.cc:5495] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 274079640ce544d8a9b865326fd90710 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "274079640ce544d8a9b865326fd90710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } health_report { overall_health: UNKNOWN } } }
I20260924 17:41:15.037266 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.034809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.98.3:46203)
0924 17:41:15.034873 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.034875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.034911 (+ 36us) server_negotiation.cc:407] Connection header received
0924 17:41:15.034941 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.034942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.034961 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.035035 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:15.035198 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.035665 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.036437 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.036627 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.036709 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.036914 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:15.037017 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:15.037107 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.037131 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.037154 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:41:15.039143 14400 raft_consensus.cc:1261] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Refusing update from remote peer 274079640ce544d8a9b865326fd90710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:15.039187 14266 raft_consensus.cc:1261] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Refusing update from remote peer 274079640ce544d8a9b865326fd90710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:15.039485 14605 consensus_queue.cc:1035] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:15.039649 14607 consensus_queue.cc:1035] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:15.042093 14610 mvcc.cc:205] Tried to move back new op lower bound from 7332952780957298688 to 7332952780915720192. Current Snapshot: MvccSnapshot[applied={T|T < 7332952780957298688}]
I20260924 17:41:15.042796 14612 mvcc.cc:205] Tried to move back new op lower bound from 7332952780957298688 to 7332952780915720192. Current Snapshot: MvccSnapshot[applied={T|T < 7332952780957298688 or (T in {7332952780957298688})}]
I20260924 17:41:15.042919 14611 mvcc.cc:205] Tried to move back new op lower bound from 7332952780957298688 to 7332952780915720192. Current Snapshot: MvccSnapshot[applied={T|T < 7332952780957298688}]
I20260924 17:41:15.044538 14120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:15.044660 14120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:15.045476 14120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Sending DeleteTablet for 3 replicas of tablet fee87946ce9d441f8e92c2b6c49b0cc7
I20260924 17:41:15.045838 14514 tablet_service.cc:1514] Processing DeleteTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59640
I20260924 17:41:15.045845 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44290
I20260924 17:41:15.045940 14246 tablet_service.cc:1514] Processing DeleteTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60202
I20260924 17:41:15.046164 14622 tablet_replica.cc:330] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: stopping tablet replica
I20260924 17:41:15.046190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14183
I20260924 17:41:15.048010 14620 tablet_replica.cc:330] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: stopping tablet replica
I20260924 17:41:15.048156 14620 raft_consensus.cc:2229] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:15.048277 14620 raft_consensus.cc:2258] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.048456 14621 tablet_replica.cc:330] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: stopping tablet replica
I20260924 17:41:15.048560 14621 raft_consensus.cc:2229] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:41:15.048661 14621 raft_consensus.cc:2258] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.049090 14620 ts_tablet_manager.cc:1905] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.050655 14621 ts_tablet_manager.cc:1905] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.050673 14620 ts_tablet_manager.cc:1918] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:15.050855 14620 log.cc:1196] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/fee87946ce9d441f8e92c2b6c49b0cc7
I20260924 17:41:15.051252 14620 ts_tablet_manager.cc:1939] T fee87946ce9d441f8e92c2b6c49b0cc7 P 827d464350014249b12b51714b25444a: Deleting consensus metadata
W20260924 17:41:15.052042 14107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34385 (error 108)
I20260924 17:41:15.052364 14108 catalog_manager.cc:4841] TS 827d464350014249b12b51714b25444a (127.0.98.2:40539): tablet fee87946ce9d441f8e92c2b6c49b0cc7 (table test-table [id=b03089f4fad945bdaea56f45fba8922f]) successfully deleted
I20260924 17:41:15.052443 14621 ts_tablet_manager.cc:1918] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:15.052511 14621 log.cc:1196] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/fee87946ce9d441f8e92c2b6c49b0cc7
W20260924 17:41:15.052701 14591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:41:15.052472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.0.1:60206)
0924 17:41:15.052523 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.052536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.052581 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.052648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34385: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 17:41:15.052805 14621 ts_tablet_manager.cc:1939] T fee87946ce9d441f8e92c2b6c49b0cc7 P 274079640ce544d8a9b865326fd90710: Deleting consensus metadata
I20260924 17:41:15.052944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14317
W20260924 17:41:15.052965 14107 catalog_manager.cc:4584] TS 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fee87946ce9d441f8e92c2b6c49b0cc7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34385: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 17:41:15.053972 14107 catalog_manager.cc:4841] TS 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): tablet fee87946ce9d441f8e92c2b6c49b0cc7 (table test-table [id=b03089f4fad945bdaea56f45fba8922f]) successfully deleted
I20260924 17:41:15.058360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14451
I20260924 17:41:15.063292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14089
I20260924 17:41:15.068768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--webserver_interface=127.0.98.62
--webserver_port=37567
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:15.153661 14624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:15.153990 14624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:15.154034 14624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:15.155296 14624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:15.155365 14624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:15.155390 14624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:15.155412 14624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:15.155439 14624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:15.156924 14624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:15.157204 14624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:15.157435 14624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:15.166074 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.166074 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.166224 14624 server_base.cc:1034] running on GCE node
W20260924 17:41:15.166380 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.166679 14624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:15.167007 14624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:15.168183 14624 hybrid_clock.cc:648] HybridClock initialized: now 1790271675168166 us; error 30 us; skew 500 ppm
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:15.170825 14624 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:15.171289 14624 webserver.cc:460] Webserver started at http://127.0.98.62:37567/ using document root <none> and password file <none>
I20260924 17:41:15.171489 14624 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:15.171540 14624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:15.172889 14624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:15.173554 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.173709 14624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:41:15.173807 14624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0097c6a2676d4ac3828cce57a61de4b0"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "f0969d191c0f063a3b766af683d6554d"
server_key_iv: "c7f6e0dc6f6e3a9e81fe2fdaa9e4aab1"
server_key_version: "encryptionkey@0"
I20260924 17:41:15.174216 14624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:15.186236 14624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:15.188491 14624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:15.188645 14624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:15.192582 14624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42383
I20260924 17:41:15.192665 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42383 every 8 connection(s)
I20260924 17:41:15.193037 14624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:15.194263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14624
I20260924 17:41:15.194564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34385
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=40591
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:15.194800 14693 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:41:15.195602 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap starting.
I20260924 17:41:15.204463 14693 log.cc:824] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:15.206535 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:15.206847 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap complete.
I20260924 17:41:15.209353 14693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.209596 14693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.209764 14693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.209829 14693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:15.209883 14693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:15.209935 14693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:41:15.210958 14693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.211105 14693 leader_election.cc:302] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0097c6a2676d4ac3828cce57a61de4b0; no voters:
I20260924 17:41:15.211305 14693 leader_election.cc:288] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:41:15.211375 14697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:41:15.211505 14697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 LEADER]: Becoming Leader. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Running, Role: LEADER
I20260924 17:41:15.211586 14693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:15.211587 14697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.212580 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0097c6a2676d4ac3828cce57a61de4b0. Latest consensus state: current_term: 2 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:41:15.212670 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:15.212903 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:41:15.212972 14709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:15.213053 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:15.213995 14709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:15.214118 14709 catalog_manager.cc:1259] Loaded cluster ID: c17d876bcc93495db5709fd129f8e536
I20260924 17:41:15.214170 14709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:15.214381 14709 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 17:41:15.217273 14709 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 17:41:15.219540 14709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:15.219787 14709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Loaded TSK: 0
I20260924 17:41:15.220093 14709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:41:15.289345 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:15.289640 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:15.289685 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:15.290906 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:15.290972 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:15.291018 14695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:15.292527 14695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40591
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:15.292797 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:15.293040 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:15.298998 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.299001 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.299000 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.299437 14695 server_base.cc:1034] running on GCE node
I20260924 17:41:15.299642 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:15.299939 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:15.301122 14695 hybrid_clock.cc:648] HybridClock initialized: now 1790271675301103 us; error 33 us; skew 500 ppm
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:15.304322 14695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:15.304921 14695 webserver.cc:460] Webserver started at http://127.0.98.1:40591/ using document root <none> and password file <none>
I20260924 17:41:15.305204 14695 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:15.305296 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:15.307245 14695 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:41:15.308022 14730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.308202 14695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:15.308306 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "9723b24fbd53484098db4aea9a5a6065"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "72ed21d1981f4dc583e8cffa30592f24"
server_key_iv: "c66e48111e25c0a780ef167d973165b9"
server_key_version: "encryptionkey@0"
I20260924 17:41:15.308676 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:15.319833 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:15.322192 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:15.322309 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:15.322587 14695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:15.323060 14737 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:41:15.325080 14695 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:41:15.325169 14695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:41:15.325218 14695 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:41:15.325886 14695 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:41:15.325945 14695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:15.326021 14737 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Bootstrap starting.
I20260924 17:41:15.333304 14695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34385
I20260924 17:41:15.333709 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:15.335265 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34385 every 8 connection(s)
I20260924 17:41:15.337179 14737 log.cc:824] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:15.340029 14737 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:15.340323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14695
I20260924 17:41:15.340394 14737 tablet_bootstrap.cc:492] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Bootstrap complete.
I20260924 17:41:15.340621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40539
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=45665
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:15.342222 14737 ts_tablet_manager.cc:1397] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:15.343957 14737 raft_consensus.cc:348] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.345077 14737 raft_consensus.cc:729] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9723b24fbd53484098db4aea9a5a6065, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.345317 14737 consensus_queue.cc:260] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.345674 14737 ts_tablet_manager.cc:1428] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 17:41:15.347625 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.334562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:58687)
0924 17:41:15.335541 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.335574 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.336027 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.336571 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.336575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.336875 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:15.337077 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.337083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.338703 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.338704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.339965 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.339968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.340973 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.340974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.341115 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.341744 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:15.341757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:15.343635 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:15.346073 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.346079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.346081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.346309 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.346586 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.346586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.346587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.346697 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.347137 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:15.347144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:15.347284 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.347360 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.347485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":118,"threads_started":1}
I20260924 17:41:15.348307 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.334631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58687 (local address 127.0.98.62:42383)
0924 17:41:15.336188 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.336196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.336223 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:41:15.336340 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.336344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.336385 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.336519 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:15.337238 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.338559 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.340069 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.340754 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.340860 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.343775 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:15.343780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:15.343781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:15.343795 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:15.345971 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.346404 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.346407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.346407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.346506 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.346767 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.346768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.346768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.347011 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:15.347899 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.347975 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.348110 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1513,"thread_start_us":67,"threads_started":1}
I20260924 17:41:15.349114 14845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:15.349226 14845 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:15.349397 14845 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:15.350108 14657 ts_manager.cc:194] Registered new tserver with Master: 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:15.351236 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58687
I20260924 17:41:15.352063 14845 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
I20260924 17:41:15.358482 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.350913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.98.1:34385)
0924 17:41:15.355361 (+ 4448us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.355365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.355372 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:41:15.355398 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.355399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.355410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.355491 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.355675 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.356337 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.357354 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.357814 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.357895 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.357958 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.358370 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.358410 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4422,"thread_start_us":64,"threads_started":1}
I20260924 17:41:15.358500 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.350670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.0.1:60216)
0924 17:41:15.355158 (+ 4488us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.355192 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.355246 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.355496 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.355499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.355501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.355599 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.355601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.356419 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.356420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.357248 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.357251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.357895 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.357896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.358264 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.358273 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.358355 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.358411 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4445,"thread_start_us":87,"threads_started":1}
I20260924 17:41:15.359122 14779 tablet_service.cc:1514] Processing DeleteTablet for tablet fee87946ce9d441f8e92c2b6c49b0cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260924 17:41:15.359649 14854 tablet_replica.cc:330] stopping tablet replica
I20260924 17:41:15.359767 14854 raft_consensus.cc:2229] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:15.359846 14854 raft_consensus.cc:2258] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.360158 14854 ts_tablet_manager.cc:1905] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.362111 14854 ts_tablet_manager.cc:1918] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:15.362200 14854 log.cc:1196] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/fee87946ce9d441f8e92c2b6c49b0cc7
I20260924 17:41:15.362445 14854 ts_tablet_manager.cc:1939] T fee87946ce9d441f8e92c2b6c49b0cc7 P 9723b24fbd53484098db4aea9a5a6065: Deleting consensus metadata
I20260924 17:41:15.362942 14643 catalog_manager.cc:4841] TS 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385): tablet fee87946ce9d441f8e92c2b6c49b0cc7 (table test-table [id=b03089f4fad945bdaea56f45fba8922f]) successfully deleted
W20260924 17:41:15.432844 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:15.433066 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:15.433094 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:15.434221 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:15.434262 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:15.434288 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:15.435806 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45665
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:15.436004 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:15.436256 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:15.441372 14862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.441358 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.441354 14860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.441620 14850 server_base.cc:1034] running on GCE node
I20260924 17:41:15.441804 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:15.442049 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:15.443183 14850 hybrid_clock.cc:648] HybridClock initialized: now 1790271675443171 us; error 29 us; skew 500 ppm
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:15.445605 14850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:15.446095 14850 webserver.cc:460] Webserver started at http://127.0.98.2:45665/ using document root <none> and password file <none>
I20260924 17:41:15.446324 14850 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:15.446413 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:15.447860 14850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:15.448450 14869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.448619 14850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:41:15.448716 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "827d464350014249b12b51714b25444a"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "89235eabd19761481f32320b650b12c3"
server_key_iv: "a4e70a46a8c98bba1c7bc5886fcf56a4"
server_key_version: "encryptionkey@0"
I20260924 17:41:15.449079 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:15.460889 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:15.463186 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:15.463353 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:15.463599 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:15.463943 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:15.463996 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.464040 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:15.464075 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.470090 14850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40539
I20260924 17:41:15.470148 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40539 every 8 connection(s)
I20260924 17:41:15.470518 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:41:15.476845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14850
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:15.477149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46203
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=41397
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:15.480676 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.471354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55993 (local address 127.0.98.62:42383)
0924 17:41:15.471417 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.471431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.471635 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:41:15.472175 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.472176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.472192 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.472251 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:15.473256 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.473789 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.474825 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.475089 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.475175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.477437 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:15.477438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:15.477438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:15.477453 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:15.479169 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.479583 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.479584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.479585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.479651 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.479880 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.479881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.479882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.480046 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:15.480120 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.480421 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.480530 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:41:15.480702 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.471318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:55993)
0924 17:41:15.471497 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.471545 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.472062 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.472342 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.472346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.472934 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:15.473135 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.473140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.473887 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.473888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.474722 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.474724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.475186 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.475187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.475325 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.475821 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:15.475829 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:15.477314 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:15.479252 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.479259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.479261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.479504 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.479719 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.479719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.479720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.479820 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.480134 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:15.480141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:15.480273 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.480383 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.480527 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260924 17:41:15.481297 14983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:15.481418 14983 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:15.481643 14983 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:15.482172 14657 ts_manager.cc:194] Registered new tserver with Master: 827d464350014249b12b51714b25444a (127.0.98.2:40539)
I20260924 17:41:15.482563 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55993
W20260924 17:41:15.560022 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:15.560228 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:15.560251 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:15.561396 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:15.561436 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:15.561462 14986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:15.562959 14986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41397
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:15.563225 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:15.563450 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:41:15.568322 14986 server_base.cc:1034] running on GCE node
W20260924 17:41:15.568325 14991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.568400 14994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.568534 14992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.568748 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:15.568941 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:15.570101 14986 hybrid_clock.cc:648] HybridClock initialized: now 1790271675570094 us; error 40 us; skew 500 ppm
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:15.572592 14986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:15.572974 14986 webserver.cc:460] Webserver started at http://127.0.98.3:41397/ using document root <none> and password file <none>
I20260924 17:41:15.573170 14986 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:15.573221 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:15.574512 14986 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:15.575021 15001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.575213 14986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.575297 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "274079640ce544d8a9b865326fd90710"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "5ed81c843bf505a94b141c41e86e521b"
server_key_iv: "257a44205aaf7bda8676df3ab6940d33"
server_key_version: "encryptionkey@0"
I20260924 17:41:15.575639 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:15.597077 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:15.599109 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:15.599261 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:15.599502 14986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:15.599813 14986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:15.599862 14986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.599906 14986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:15.599933 14986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.605484 14986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46203
I20260924 17:41:15.605561 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46203 every 8 connection(s)
I20260924 17:41:15.605899 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:15.613037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 14986
I20260924 17:41:15.615751 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.606635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53283 (local address 127.0.98.62:42383)
0924 17:41:15.606684 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.606687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.606917 (+ 230us) server_negotiation.cc:407] Connection header received
0924 17:41:15.607396 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.607397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.607408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.607459 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:15.608411 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.608892 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.609877 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.610053 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.610103 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.612542 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:15.612544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:15.612544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:15.612551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:15.614266 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.614801 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.614802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.614802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.614851 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.615139 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.615140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.615140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.615241 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:15.615292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.615574 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.615654 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:41:15.615904 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.606643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:53283)
0924 17:41:15.606800 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.606837 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.607307 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.607532 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.607536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.608112 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:15.608305 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.608326 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.608969 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.608970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.609804 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.609807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.610138 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.610139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.610255 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.610779 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:15.610787 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:15.612461 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:15.614350 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.614355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.614357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.614719 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.614893 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:15.614894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:15.614895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:15.615060 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:15.615297 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:15.615302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:15.615427 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.615508 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.615628 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260924 17:41:15.616351 15115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260924 17:41:15.616456 15115 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:15.616644 15115 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260924 17:41:15.616971 14657 ts_manager.cc:194] Registered new tserver with Master: 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
I20260924 17:41:15.617446 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53283
I20260924 17:41:15.621163 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.613624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.98.62:42383)
0924 17:41:15.614341 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.614342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.614348 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:41:15.614362 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.614362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.614371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.614415 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:15.615311 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.615989 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.617045 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.617327 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.617393 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.618717 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:15.618718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:15.618718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:15.618724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:15.620004 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.620359 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.620361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.620361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.620421 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:15.620617 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:15.620618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:15.620618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:15.620730 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:15.620935 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.620973 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.621056 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260924 17:41:15.621836 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:15.624677 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.622165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.0.98.62:42383)
0924 17:41:15.622223 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.622225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.622295 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:41:15.622380 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.622380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.622418 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.622501 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:15.622617 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.623155 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.623905 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.624117 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.624184 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.624360 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:15.624466 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:15.624560 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.624578 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.624601 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:41:15.624962 14657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:41:15.626525 14657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:41:15.628888 14779 tablet_service.cc:1467] Processing CreateTablet for tablet ee75a262f67f401fb12dee0cf7e1329f (DEFAULT_TABLE table=test-table [id=b1f4e25adf664537bf9e2befb3494220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:15.629119 14779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee75a262f67f401fb12dee0cf7e1329f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:15.630729 14737 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Bootstrap starting.
I20260924 17:41:15.631078 14737 tablet_bootstrap.cc:654] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:15.631597 14737 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: No bootstrap required, opened a new log
I20260924 17:41:15.631659 14737 ts_tablet_manager.cc:1397] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:15.631778 14737 raft_consensus.cc:348] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.631846 14737 raft_consensus.cc:374] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:15.631892 14737 raft_consensus.cc:729] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9723b24fbd53484098db4aea9a5a6065, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.631946 14737 consensus_queue.cc:260] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.632031 14737 ts_tablet_manager.cc:1428] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:41:15.632064 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.628411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.0.1:44306)
0924 17:41:15.628466 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.628478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.628523 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.628986 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.628986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.628988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.629037 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.629038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.630016 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.630017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.630854 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.630856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.631787 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.631789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.631858 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.631863 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.631935 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.631985 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:41:15.632073 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.628624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.0.98.2:40539)
0924 17:41:15.628828 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.628833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.628844 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:41:15.628887 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.628890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.628907 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.628989 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.629095 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.629905 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.630963 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.631719 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.631788 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.631855 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.631950 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.631991 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260924 17:41:15.632839 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.628466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.0.1:59656)
0924 17:41:15.628614 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.628622 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.628686 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.629278 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.629278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.629281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.629315 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.629316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.630603 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.630603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.631678 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.631681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.632644 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.632645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.632709 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.632713 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.632758 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.632789 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260924 17:41:15.632913 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.628760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.98.3:46203)
0924 17:41:15.628941 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.628945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.628955 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:15.629098 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.629100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.629117 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.629280 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.629380 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.630487 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.631827 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.632540 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.632614 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.632668 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.632791 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.632838 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260924 17:41:15.632792 14917 tablet_service.cc:1467] Processing CreateTablet for tablet ee75a262f67f401fb12dee0cf7e1329f (DEFAULT_TABLE table=test-table [id=b1f4e25adf664537bf9e2befb3494220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:15.633136 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee75a262f67f401fb12dee0cf7e1329f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:15.633574 15049 tablet_service.cc:1467] Processing CreateTablet for tablet ee75a262f67f401fb12dee0cf7e1329f (DEFAULT_TABLE table=test-table [id=b1f4e25adf664537bf9e2befb3494220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:15.633919 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee75a262f67f401fb12dee0cf7e1329f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:15.636340 15125 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Bootstrap starting.
I20260924 17:41:15.636657 15126 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Bootstrap starting.
I20260924 17:41:15.636957 15125 tablet_bootstrap.cc:654] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:15.637271 15126 tablet_bootstrap.cc:654] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:15.637475 15125 log.cc:824] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:15.637841 15126 log.cc:824] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:15.638222 15125 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: No bootstrap required, opened a new log
I20260924 17:41:15.638296 15125 ts_tablet_manager.cc:1397] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:15.638550 15126 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: No bootstrap required, opened a new log
I20260924 17:41:15.638622 15126 ts_tablet_manager.cc:1397] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:15.639932 15125 raft_consensus.cc:348] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.640080 15125 raft_consensus.cc:374] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:15.640103 15125 raft_consensus.cc:729] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827d464350014249b12b51714b25444a, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.640233 15125 consensus_queue.cc:260] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.640139 15126 raft_consensus.cc:348] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.640316 15126 raft_consensus.cc:374] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:15.640364 15126 raft_consensus.cc:729] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274079640ce544d8a9b865326fd90710, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.640476 15126 consensus_queue.cc:260] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.640530 14983 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
I20260924 17:41:15.640576 15125 ts_tablet_manager.cc:1428] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:41:15.640744 15126 ts_tablet_manager.cc:1428] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:15.640828 15115 heartbeater.cc:503] Master 127.0.98.62:42383 was elected leader, sending a full tablet report...
W20260924 17:41:15.721146 14984 tablet.cc:2369] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:15.723757 15129 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:15.723867 15129 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.724258 15129 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203), 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:15.727221 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.724503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52061 (local address 127.0.98.3:46203)
0924 17:41:15.724545 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.724547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.724558 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:41:15.724658 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.724658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.724670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.724748 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.724928 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.725476 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.726446 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.726843 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.726899 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.726908 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.727098 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.727145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:41:15.727226 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.724462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.98.2:52061)
0924 17:41:15.724515 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.724527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.724572 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.724765 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.724766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.724767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.724852 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.724854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.725567 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.725568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.726360 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.726362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.726924 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.726926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.727013 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.727016 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.727092 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.727137 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:41:15.727305 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.724666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47531 (local address 127.0.98.1:34385)
0924 17:41:15.724730 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.724732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.724852 (+ 120us) server_negotiation.cc:407] Connection header received
0924 17:41:15.724902 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.724903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.724916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.724967 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:15.725115 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.725608 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.726525 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.726898 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.727006 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.727017 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.727192 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.727233 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:41:15.727522 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.724597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.98.2:47531)
0924 17:41:15.724785 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.724794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:15.724856 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:15.724968 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:15.724968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:15.724970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:15.725031 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.725033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.725695 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.725696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.726446 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:15.726448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.727011 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.727012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:15.727119 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.727123 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:15.727178 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:41:15.727216 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260924 17:41:15.727612 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee75a262f67f401fb12dee0cf7e1329f" candidate_uuid: "827d464350014249b12b51714b25444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274079640ce544d8a9b865326fd90710" is_pre_election: true
I20260924 17:41:15.727773 15069 raft_consensus.cc:2454] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827d464350014249b12b51714b25444a in term 0.
I20260924 17:41:15.727998 14872 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a; no voters:
I20260924 17:41:15.728154 15129 raft_consensus.cc:2781] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:15.728099 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee75a262f67f401fb12dee0cf7e1329f" candidate_uuid: "827d464350014249b12b51714b25444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9723b24fbd53484098db4aea9a5a6065" is_pre_election: true
I20260924 17:41:15.728204 15129 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:15.728227 15129 raft_consensus.cc:3037] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.728255 14794 raft_consensus.cc:2454] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827d464350014249b12b51714b25444a in term 0.
I20260924 17:41:15.728960 15129 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.729069 15129 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [CANDIDATE]: Term 1 election: Requested vote from peers 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203), 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385)
I20260924 17:41:15.729285 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee75a262f67f401fb12dee0cf7e1329f" candidate_uuid: "827d464350014249b12b51714b25444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274079640ce544d8a9b865326fd90710"
I20260924 17:41:15.729336 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee75a262f67f401fb12dee0cf7e1329f" candidate_uuid: "827d464350014249b12b51714b25444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9723b24fbd53484098db4aea9a5a6065"
I20260924 17:41:15.729363 15069 raft_consensus.cc:3037] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.729449 14794 raft_consensus.cc:3037] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:15.730074 15069 raft_consensus.cc:2454] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827d464350014249b12b51714b25444a in term 1.
I20260924 17:41:15.730230 14794 raft_consensus.cc:2454] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827d464350014249b12b51714b25444a in term 1.
I20260924 17:41:15.730237 14872 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a; no voters:
I20260924 17:41:15.730335 15129 raft_consensus.cc:2781] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:15.730500 15129 raft_consensus.cc:686] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 1 LEADER]: Becoming Leader. State: Replica: 827d464350014249b12b51714b25444a, State: Running, Role: LEADER
I20260924 17:41:15.730610 15129 consensus_queue.cc:237] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:15.731479 14657 catalog_manager.cc:5495] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a reported cstate change: term changed from 0 to 1, leader changed from <none> to 827d464350014249b12b51714b25444a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "827d464350014249b12b51714b25444a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } health_report { overall_health: UNKNOWN } } }
I20260924 17:41:15.750939 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.748846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.0.98.2:40539)
0924 17:41:15.748906 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:15.748908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:15.748917 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:41:15.748956 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:15.748957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:15.748974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:15.749046 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:15.749152 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.749607 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.750280 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:15.750457 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:15.750506 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:15.750632 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:15.750729 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:15.750785 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:15.750814 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:41:15.750831 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:41:15.752913 14794 raft_consensus.cc:1261] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Refusing update from remote peer 827d464350014249b12b51714b25444a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:15.752895 15069 raft_consensus.cc:1261] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Refusing update from remote peer 827d464350014249b12b51714b25444a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:15.753242 15129 consensus_queue.cc:1035] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [LEADER]: Connected to new peer: Peer: permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:15.753388 15132 consensus_queue.cc:1035] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:15.755818 15137 mvcc.cc:205] Tried to move back new op lower bound from 7332952783880609792 to 7332952783793033216. Current Snapshot: MvccSnapshot[applied={T|T < 7332952783880609792}]
I20260924 17:41:15.756619 15136 mvcc.cc:205] Tried to move back new op lower bound from 7332952783880609792 to 7332952783793033216. Current Snapshot: MvccSnapshot[applied={T|T < 7332952783880609792 or (T in {7332952783880609792})}]
I20260924 17:41:15.756750 15135 mvcc.cc:205] Tried to move back new op lower bound from 7332952783880609792 to 7332952783793033216. Current Snapshot: MvccSnapshot[applied={T|T < 7332952783880609792 or (T in {7332952783880609792})}]
I20260924 17:41:15.757761 14656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:15.757900 14656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:15.758679 14656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Sending DeleteTablet for 3 replicas of tablet ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:15.759017 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet ee75a262f67f401fb12dee0cf7e1329f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44306
I20260924 17:41:15.759025 14779 tablet_service.cc:1514] Processing DeleteTablet for tablet ee75a262f67f401fb12dee0cf7e1329f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260924 17:41:15.759066 15049 tablet_service.cc:1514] Processing DeleteTablet for tablet ee75a262f67f401fb12dee0cf7e1329f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59656
I20260924 17:41:15.759157 14854 tablet_replica.cc:330] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: stopping tablet replica
I20260924 17:41:15.759203 14854 raft_consensus.cc:2229] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:15.759255 14854 raft_consensus.cc:2258] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.759402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14695
I20260924 17:41:15.759579 14854 ts_tablet_manager.cc:1905] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.760023 15145 tablet_replica.cc:330] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: stopping tablet replica
I20260924 17:41:15.760120 15145 raft_consensus.cc:2229] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:15.760221 15145 raft_consensus.cc:2258] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.760314 15146 tablet_replica.cc:330] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: stopping tablet replica
I20260924 17:41:15.760396 15146 raft_consensus.cc:2229] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:41:15.760492 15146 raft_consensus.cc:2258] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:15.760916 15146 ts_tablet_manager.cc:1905] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.762893 15146 ts_tablet_manager.cc:1918] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:15.762972 15146 log.cc:1196] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:15.763284 15146 ts_tablet_manager.cc:1939] T ee75a262f67f401fb12dee0cf7e1329f P 827d464350014249b12b51714b25444a: Deleting consensus metadata
I20260924 17:41:15.763872 14644 catalog_manager.cc:4841] TS 827d464350014249b12b51714b25444a (127.0.98.2:40539): tablet ee75a262f67f401fb12dee0cf7e1329f (table test-table [id=b1f4e25adf664537bf9e2befb3494220]) successfully deleted
I20260924 17:41:15.764096 15145 ts_tablet_manager.cc:1905] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:15.765956 15145 ts_tablet_manager.cc:1918] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:15.766036 15145 log.cc:1196] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:15.766315 15145 ts_tablet_manager.cc:1939] T ee75a262f67f401fb12dee0cf7e1329f P 274079640ce544d8a9b865326fd90710: Deleting consensus metadata
W20260924 17:41:15.766692 14643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34385 (error 108)
I20260924 17:41:15.767099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14850
I20260924 17:41:15.767205 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:15.767074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34385 (local address 127.0.0.1:60226)
0924 17:41:15.767120 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:15.767164 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 17:41:15.767395 14643 catalog_manager.cc:4584] TS 9723b24fbd53484098db4aea9a5a6065 (127.0.98.1:34385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee75a262f67f401fb12dee0cf7e1329f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34385: connect: Connection refused (error 111)
I20260924 17:41:15.767901 14643 catalog_manager.cc:4841] TS 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): tablet ee75a262f67f401fb12dee0cf7e1329f (table test-table [id=b1f4e25adf664537bf9e2befb3494220]) successfully deleted
I20260924 17:41:15.772601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14986
I20260924 17:41:15.777247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 14624
I20260924 17:41:15.782425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--webserver_interface=127.0.98.62
--webserver_port=37567
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:15.865542 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:15.865756 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:15.865777 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:15.866887 15148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:15.866927 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:15.866942 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:15.866954 15148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:15.866966 15148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:15.868567 15148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:15.868775 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:15.868954 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:15.873826 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:15.873917 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.874002 15148 server_base.cc:1034] running on GCE node
W20260924 17:41:15.873921 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:15.874377 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:15.874668 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:15.875816 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790271675875796 us; error 38 us; skew 500 ppm
Sep 24 17:41:15 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:15.878269 15148 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:15.878719 15148 webserver.cc:460] Webserver started at http://127.0.98.62:37567/ using document root <none> and password file <none>
I20260924 17:41:15.878940 15148 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:15.878988 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:15.880362 15148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:15.881072 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:15.881249 15148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:15.881335 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0097c6a2676d4ac3828cce57a61de4b0"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "f0969d191c0f063a3b766af683d6554d"
server_key_iv: "c7f6e0dc6f6e3a9e81fe2fdaa9e4aab1"
server_key_version: "encryptionkey@0"
I20260924 17:41:15.881692 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:15.904876 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:15.907064 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:15.907229 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:15.911233 15148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42383
I20260924 17:41:15.911285 15215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42383 every 8 connection(s)
I20260924 17:41:15.911645 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:15.913174 15216 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:41:15.913843 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap starting.
I20260924 17:41:15.917346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15148
I20260924 17:41:15.917646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34385
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=40591
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:15.922683 15216 log.cc:824] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:15.924973 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:15.925297 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0: Bootstrap complete.
I20260924 17:41:15.927646 15216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.927892 15216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Initialized, Role: FOLLOWER
I20260924 17:41:15.928076 15216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.928161 15216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:15.928213 15216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:15.928261 15216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 2 FOLLOWER]: Advancing to term 3
I20260924 17:41:15.929276 15216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.929395 15216 leader_election.cc:302] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0097c6a2676d4ac3828cce57a61de4b0; no voters:
I20260924 17:41:15.929593 15216 leader_election.cc:288] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 17:41:15.929628 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 17:41:15.929742 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [term 3 LEADER]: Becoming Leader. State: Replica: 0097c6a2676d4ac3828cce57a61de4b0, State: Running, Role: LEADER
I20260924 17:41:15.929828 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260924 17:41:15.929869 15216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:15.930091 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0097c6a2676d4ac3828cce57a61de4b0. Latest consensus state: current_term: 3 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:41:15.930167 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:15.930097 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0097c6a2676d4ac3828cce57a61de4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097c6a2676d4ac3828cce57a61de4b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260924 17:41:15.930665 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097c6a2676d4ac3828cce57a61de4b0 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:15.931313 15224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:15.932183 15224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:15.932325 15224 catalog_manager.cc:1259] Loaded cluster ID: c17d876bcc93495db5709fd129f8e536
I20260924 17:41:15.932361 15224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 17:41:15.932564 15224 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f94d5aa29ad google::LogMessage::Fail() at ??:0
@ 0x7f94d5aa588a google::LogMessage::SendToLog() at ??:0
@ 0x7f94d5aa24de google::LogMessage::Flush() at ??:0
@ 0x7f94d5aa3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f94dd9c30f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f94dd9cc8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f94d79700d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f94d7965954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f94d87c96db start_thread at ??:0
@ 0x7f94d45a061f clone at ??:0
W20260924 17:41:16.018869 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:16.019157 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:16.019222 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:16.021057 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:16.021124 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:16.021170 15218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:16.023558 15218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40591
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:16.023839 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:16.024117 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:16.029692 15252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:16.029716 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.029798 15218 server_base.cc:1034] running on GCE node
W20260924 17:41:16.029714 15255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.030071 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:16.030316 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:16.031483 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790271676031465 us; error 44 us; skew 500 ppm
Sep 24 17:41:16 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:16.034291 15218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:16.034818 15218 webserver.cc:460] Webserver started at http://127.0.98.1:40591/ using document root <none> and password file <none>
I20260924 17:41:16.035135 15218 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:16.035200 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:16.036684 15218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:16.037501 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.037755 15218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:16.037855 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "9723b24fbd53484098db4aea9a5a6065"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "72ed21d1981f4dc583e8cffa30592f24"
server_key_iv: "c66e48111e25c0a780ef167d973165b9"
server_key_version: "encryptionkey@0"
I20260924 17:41:16.038249 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:16.053424 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:16.056190 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:16.056367 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:16.056667 15218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:16.057240 15269 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:41:16.058753 15218 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:41:16.058818 15218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:41:16.058866 15218 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:41:16.059810 15218 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:41:16.059868 15218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:16.059976 15269 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Bootstrap starting.
I20260924 17:41:16.061707 15269 log.cc:824] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:16.065120 15269 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:16.065430 15269 tablet_bootstrap.cc:492] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Bootstrap complete.
I20260924 17:41:16.065546 15269 ts_tablet_manager.cc:1397] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 17:41:16.067394 15218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34385
I20260924 17:41:16.067517 15377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34385 every 8 connection(s)
I20260924 17:41:16.067819 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:16.067636 15269 raft_consensus.cc:348] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:16.067997 15269 raft_consensus.cc:729] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9723b24fbd53484098db4aea9a5a6065, State: Initialized, Role: FOLLOWER
I20260924 17:41:16.068204 15269 consensus_queue.cc:260] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:16.068899 15269 ts_tablet_manager.cc:1428] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:41:16.075989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15218
I20260924 17:41:16.076299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40539
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=45665
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:16.140616 15381 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:41:16.068623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:46047)
0924 17:41:16.069069 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.069112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:16.069257 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:16.140407 (+ 71150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":177,"threads_started":1}
W20260924 17:41:16.140929 15378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42383: Client connection negotiation failed: client connection to 127.0.98.62:42383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:41:16.141325 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.141184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:59725)
0924 17:41:16.141244 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.141294 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:41:16.141688 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.141585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:51131)
0924 17:41:16.141632 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.141663 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:41:16.141829 15378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:41:16.161077 15382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:16.161347 15382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:16.161394 15382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:16.162613 15382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:16.162684 15382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:16.162729 15382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:16.164285 15382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45665
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:16.164566 15382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:16.164803 15382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:16.169866 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.169937 15382 server_base.cc:1034] running on GCE node
W20260924 17:41:16.169981 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:16.170042 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.170325 15382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:16.170572 15382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:16.171722 15382 hybrid_clock.cc:648] HybridClock initialized: now 1790271676171707 us; error 27 us; skew 500 ppm
Sep 24 17:41:16 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:16.174374 15382 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:16.174875 15382 webserver.cc:460] Webserver started at http://127.0.98.2:45665/ using document root <none> and password file <none>
I20260924 17:41:16.175161 15382 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:16.175228 15382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:16.176604 15382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:16.177273 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.177472 15382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:16.177578 15382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "827d464350014249b12b51714b25444a"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "89235eabd19761481f32320b650b12c3"
server_key_iv: "a4e70a46a8c98bba1c7bc5886fcf56a4"
server_key_version: "encryptionkey@0"
I20260924 17:41:16.177958 15382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:16.202656 15382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:16.204720 15382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:16.204900 15382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:16.205147 15382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:16.205472 15382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:16.205529 15382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.205570 15382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:16.205600 15382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.211575 15382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40539
I20260924 17:41:16.211632 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40539 every 8 connection(s)
I20260924 17:41:16.212010 15382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:41:16.213074 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.212715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:34333)
0924 17:41:16.212865 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.212937 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260924 17:41:16.213491 15511 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42383: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
I20260924 17:41:16.213768 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.213684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:46669)
0924 17:41:16.213725 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.213751 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 17:41:16.214085 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.214016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:43451)
0924 17:41:16.214052 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.214072 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 17:41:16.214200 15511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:41:16.221736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15382
I20260924 17:41:16.222020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46203
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42383
--webserver_port=41397
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:16.307226 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:16.307449 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:16.307474 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:16.308658 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:16.308697 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:16.308724 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:16.310171 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41397
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:16.310369 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:16.310564 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:16.315907 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:16.315920 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.315989 15514 server_base.cc:1034] running on GCE node
W20260924 17:41:16.315889 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:16.316370 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:16.316650 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:16.317812 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790271676317799 us; error 35 us; skew 500 ppm
Sep 24 17:41:16 dist-test-slave-d0pf krb5kdc[14076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:16.320483 15514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:16.321019 15514 webserver.cc:460] Webserver started at http://127.0.98.3:41397/ using document root <none> and password file <none>
I20260924 17:41:16.321256 15514 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:16.321322 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:16.322726 15514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:16.323460 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.323688 15514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:16.323796 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "274079640ce544d8a9b865326fd90710"
format_stamp: "Formatted at 2026-09-24 17:41:14 on dist-test-slave-d0pf"
server_key: "5ed81c843bf505a94b141c41e86e521b"
server_key_iv: "257a44205aaf7bda8676df3ab6940d33"
server_key_version: "encryptionkey@0"
I20260924 17:41:16.324170 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:16.337986 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:16.340408 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:16.340583 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:16.340830 15514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:16.341137 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:16.341188 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.341239 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:16.341280 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:16.347263 15514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46203
I20260924 17:41:16.347393 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46203 every 8 connection(s)
I20260924 17:41:16.347725 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:16.348752 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.348390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:34259)
0924 17:41:16.348545 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.348614 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
W20260924 17:41:16.349140 15643 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42383: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
I20260924 17:41:16.349406 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.349322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:45857)
0924 17:41:16.349366 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.349390 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 17:41:16.349647 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:16.349600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:46833)
0924 17:41:16.349619 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:16.349636 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260924 17:41:16.349759 15643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:41:16.357682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15514
I20260924 17:41:17.142380 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.142165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:42525)
0924 17:41:17.142282 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:17.142318 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 17:41:17.214728 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.214498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:47627)
0924 17:41:17.214634 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:17.214671 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260924 17:41:17.350366 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.350114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:57881)
0924 17:41:17.350274 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:17.350316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260924 17:41:17.836462 15650 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:17.836609 15650 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:17.836988 15650 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
Sep 24 17:41:17 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 17:41:17 dist-test-slave-d0pf krb5kdc[14076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 17:41:17.848234 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.837178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.98.1:57609)
0924 17:41:17.837367 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:17.837377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:17.837436 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:17.838150 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:17.838153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:17.838533 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:17.838758 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:17.838764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.839937 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.839938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:17.841346 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:17.841349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.841967 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.841968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:17.842097 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:17.842692 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:17.842701 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:17.844718 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:17.846708 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:17.846709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:17.846710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:17.846930 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:17.847335 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:17.847336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:17.847336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:17.847394 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:17.847807 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:17.847807 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:17.847863 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 17:41:17.847948 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 17:41:17.848127 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":114,"thread_start_us":58,"threads_started":1}
I20260924 17:41:17.848250 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.837323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41469 (local address 127.0.98.2:40539)
0924 17:41:17.837484 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:17.837494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:17.837515 (+ 21us) server_negotiation.cc:407] Connection header received
0924 17:41:17.837932 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:17.837934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:17.837970 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:17.838117 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:17.839091 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.839969 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.841423 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.841846 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.841969 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:17.844453 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:17.844473 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:17.844473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:17.844487 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:17.846417 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:17.846893 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:17.846896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:17.846896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:17.847011 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:17.847287 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:17.847288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:17.847288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:17.847586 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:17.847646 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:17.847976 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 17:41:17.848114 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260924 17:41:17.848263 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.837239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57609 (local address 127.0.98.3:46203)
0924 17:41:17.837431 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:17.837440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:17.837473 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:41:17.837915 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:17.837917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:17.837958 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:17.838089 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:17.838882 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.839839 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.841464 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.841864 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.842027 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:17.844831 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:17.844835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:17.844835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:17.844845 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:17.846636 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:17.847140 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:17.847143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:17.847144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:17.847252 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:17.847450 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:17.847452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:17.847452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:17.847721 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:17.847818 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:17.848014 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 17:41:17.848150 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":47,"threads_started":1}
I20260924 17:41:17.848347 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:17.837126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.98.1:41469)
0924 17:41:17.837262 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:17.837272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:17.837327 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:17.838151 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:17.838154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:17.838534 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:17.838946 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:17.838948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.840105 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.840106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:17.841347 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:17.841349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:17.841942 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:17.841944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:17.842098 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:17.842735 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:17.842740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:17.844341 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:17.846502 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:17.846509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:17.846511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:17.846810 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:17.847092 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:17.847093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:17.847093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:17.847200 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:17.847652 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:17.847658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:17.847801 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 17:41:17.847909 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 17:41:17.848205 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":135,"thread_start_us":63,"threads_started":1}
W20260924 17:41:17.849630 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:17.849731 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:17.849795 15266 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:17.849896 15650 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:18.143023 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:18.142858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:51029)
0924 17:41:18.142944 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:18.142982 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 17:41:18.215615 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:18.215344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:37699)
0924 17:41:18.215508 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:18.215566 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260924 17:41:18.351137 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:18.350895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:57121)
0924 17:41:18.351007 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:18.351094 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260924 17:41:19.143772 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:19.143549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:59459)
0924 17:41:19.143665 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:19.143709 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 17:41:19.216259 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:19.216055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:54743)
0924 17:41:19.216175 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:19.216215 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260924 17:41:19.352013 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:19.351767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:36603)
0924 17:41:19.351900 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:19.351945 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 17:41:19.676795 15660 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:19.676918 15660 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:19.677117 15660 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
W20260924 17:41:19.677507 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:19.677598 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:19.677685 15265 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:19.677780 15660 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:20.144634 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:20.144295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:58101)
0924 17:41:20.144421 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:20.144547 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260924 17:41:20.216985 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:20.216750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:57173)
0924 17:41:20.216896 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:20.216942 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260924 17:41:20.352701 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:20.352472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:35795)
0924 17:41:20.352619 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:20.352660 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260924 17:41:21.145364 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:21.145158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:33583)
0924 17:41:21.145285 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:21.145318 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260924 17:41:21.217741 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:21.217505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:52691)
0924 17:41:21.217656 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:21.217699 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 17:41:21.353426 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:21.353185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:41687)
0924 17:41:21.353338 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:21.353381 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260924 17:41:21.489348 15667 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:21.489462 15667 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:21.489661 15667 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
W20260924 17:41:21.490058 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:21.490116 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:21.490147 15265 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:21.490233 15667 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:22.146154 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:22.145896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:55137)
0924 17:41:22.146052 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:22.146100 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260924 17:41:22.218678 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:22.218412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:58659)
0924 17:41:22.218567 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:22.218611 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260924 17:41:22.354180 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:22.353951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:52337)
0924 17:41:22.354092 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:22.354135 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 17:41:23.146961 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:23.146726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:40983)
0924 17:41:23.146863 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:23.146908 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 17:41:23.158113 15672 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:23.158258 15672 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:23.158458 15672 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
W20260924 17:41:23.159065 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:23.159147 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:23.159192 15265 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:23.159277 15672 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:23.219532 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:23.219294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:46003)
0924 17:41:23.219440 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:23.219486 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260924 17:41:23.354877 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:23.354660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:40901)
0924 17:41:23.354799 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:23.354834 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 17:41:24.147653 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:24.147441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:41927)
0924 17:41:24.147570 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:24.147608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260924 17:41:24.220463 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:24.220216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:52993)
0924 17:41:24.220372 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:24.220417 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260924 17:41:24.355625 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:24.355405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:39993)
0924 17:41:24.355530 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:24.355575 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260924 17:41:25.148399 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:25.148156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:49073)
0924 17:41:25.148299 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:25.148352 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 17:41:25.214020 15679 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:25.214118 15679 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:25.214351 15679 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
W20260924 17:41:25.214812 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:25.214923 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:25.214962 15265 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:25.215078 15679 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:25.221285 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:25.221028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:37741)
0924 17:41:25.221181 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:25.221229 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260924 17:41:25.356370 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:25.356128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:41181)
0924 17:41:25.356282 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:25.356324 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260924 17:41:26.149117 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:26.148898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:38713)
0924 17:41:26.149033 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:26.149069 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260924 17:41:26.221982 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:26.221730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:47603)
0924 17:41:26.221898 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:26.221937 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260924 17:41:26.357155 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:26.356901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:39857)
0924 17:41:26.357059 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:26.357108 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260924 17:41:27.149883 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:27.149646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:45607)
0924 17:41:27.149791 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:27.149837 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260924 17:41:27.222803 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:27.222554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:43279)
0924 17:41:27.222712 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:27.222757 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260924 17:41:27.357921 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:27.357677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:47375)
0924 17:41:27.357828 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:27.357870 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260924 17:41:28.150763 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:28.150499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:35385)
0924 17:41:28.150662 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:28.150712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260924 17:41:28.223630 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:28.223374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:41925)
0924 17:41:28.223545 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:28.223584 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260924 17:41:28.358634 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:28.358405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:54441)
0924 17:41:28.358545 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:28.358588 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 17:41:29.151557 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:29.151327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:60559)
0924 17:41:29.151469 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:29.151511 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260924 17:41:29.224521 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:29.224186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:46903)
0924 17:41:29.224398 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:29.224471 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260924 17:41:29.359445 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:29.359211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:37593)
0924 17:41:29.359371 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:29.359401 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260924 17:41:29.463287 15694 raft_consensus.cc:482] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:29.463397 15694 raft_consensus.cc:504] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827d464350014249b12b51714b25444a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } } peers { permanent_uuid: "274079640ce544d8a9b865326fd90710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "9723b24fbd53484098db4aea9a5a6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34385 } }
I20260924 17:41:29.463646 15694 leader_election.cc:288] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 827d464350014249b12b51714b25444a (127.0.98.2:40539), 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203)
W20260924 17:41:29.464216 15265 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274079640ce544d8a9b865326fd90710 (127.0.98.3:46203): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
W20260924 17:41:29.464329 15266 leader_election.cc:341] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827d464350014249b12b51714b25444a (127.0.98.2:40539): Not found: Tablet not found: ee75a262f67f401fb12dee0cf7e1329f
I20260924 17:41:29.464429 15266 leader_election.cc:302] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9723b24fbd53484098db4aea9a5a6065; no voters: 274079640ce544d8a9b865326fd90710, 827d464350014249b12b51714b25444a
I20260924 17:41:29.464514 15694 raft_consensus.cc:2726] T ee75a262f67f401fb12dee0cf7e1329f P 9723b24fbd53484098db4aea9a5a6065 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:41:30.152401 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:30.152146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:51761)
0924 17:41:30.152302 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:30.152348 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260924 17:41:30.225360 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:30.225108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:51843)
0924 17:41:30.225262 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:30.225311 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260924 17:41:30.360203 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:30.359943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:41973)
0924 17:41:30.360110 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:30.360156 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260924 17:41:31.153208 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:31.152974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:33847)
0924 17:41:31.153101 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:31.153140 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260924 17:41:31.226203 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:31.225988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:33495)
0924 17:41:31.226113 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:31.226148 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 17:41:31.360153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15218
I20260924 17:41:31.361624 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:31.361298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:40891)
0924 17:41:31.361530 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:31.361567 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":51,"threads_started":1}
I20260924 17:41:31.365054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15382
I20260924 17:41:31.369269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15514
2026-09-24T17:41:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18400 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:41:31 dist-test-slave-d0pf krb5kdc[15703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:41:31 dist-test-slave-d0pf krb5kdc[15703](info): set up 2 sockets
Sep 24 17:41:31 dist-test-slave-d0pf krb5kdc[15703](info): commencing operation
krb5kdc: starting...
W20260924 17:41:32.423128 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:41:32.541692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:32.620703 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:32.620939 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:32.620975 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:32.622115 15716 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:41:32.622181 15716 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:41:32.622202 15716 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:41:32.622220 15716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:32.622253 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:32.622278 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:32.622305 15716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:32.622334 15716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:32.623831 15716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:32.624111 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:32.624354 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:32.629704 15724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.629731 15716 server_base.cc:1034] running on GCE node
W20260924 17:41:32.629704 15721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:32.629734 15722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.630277 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:32.630581 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:32.631762 15716 hybrid_clock.cc:648] HybridClock initialized: now 1790271692631747 us; error 29 us; skew 500 ppm
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:32.634348 15716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:32.634847 15716 webserver.cc:460] Webserver started at http://127.0.98.62:43401/ using document root <none> and password file <none>
I20260924 17:41:32.635146 15716 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:32.635218 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:32.635375 15716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:32.636510 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "f7c320746dab4013832325a3a6c1bf50"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "16b8d28f38c0ed166cc0e4ba19da5f5f"
server_key_iv: "a961ac17fdc00b1cf0b2a4eaa3f3e12d"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.636885 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7c320746dab4013832325a3a6c1bf50"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "16b8d28f38c0ed166cc0e4ba19da5f5f"
server_key_iv: "a961ac17fdc00b1cf0b2a4eaa3f3e12d"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.638460 15716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:41:32.639380 15731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.639554 15716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:32.639669 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f7c320746dab4013832325a3a6c1bf50"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "16b8d28f38c0ed166cc0e4ba19da5f5f"
server_key_iv: "a961ac17fdc00b1cf0b2a4eaa3f3e12d"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.639750 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:32.657896 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:32.660401 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:32.660590 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:32.664832 15716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34549
I20260924 17:41:32.664868 15783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34549 every 8 connection(s)
I20260924 17:41:32.665223 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:32.666064 15784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:32.666517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15716
I20260924 17:41:32.666622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:41:32.666849 392 external_mini_cluster.cc:1428] Setting key 3c92f8a512eac73c46eace9033f07575
I20260924 17:41:32.669178 15784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap starting.
I20260924 17:41:32.669790 15784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:32.670104 15784 log.cc:824] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:32.670843 15784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: No bootstrap required, opened a new log
I20260924 17:41:32.673427 15784 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:32.673564 15784 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:32.673592 15784 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Initialized, Role: FOLLOWER
I20260924 17:41:32.673736 15784 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:32.673811 15784 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:32.673834 15784 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:32.673899 15784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:32.674501 15784 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:32.674623 15784 leader_election.cc:302] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c320746dab4013832325a3a6c1bf50; no voters:
I20260924 17:41:32.674788 15784 leader_election.cc:288] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:41:32.674820 15789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:32.674989 15789 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 LEADER]: Becoming Leader. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Running, Role: LEADER
I20260924 17:41:32.675156 15789 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:32.675487 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c320746dab4013832325a3a6c1bf50. Latest consensus state: current_term: 1 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:41:32.675591 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:32.675474 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:41:32.675717 15784 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:32.675726 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:32.676031 15793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:32.676470 15793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:32.678522 15793 catalog_manager.cc:1347] Generated new cluster ID: 719b21d9978b43a695602a0dc7367824
I20260924 17:41:32.678576 15793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:32.678869 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:32.667437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.98.62:34549)
0924 17:41:32.667608 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:32.667618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:32.667658 (+ 40us) server_negotiation.cc:407] Connection header received
0924 17:41:32.668050 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:32.668056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:32.668102 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:32.668224 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:32.669018 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.670027 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.671097 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.671509 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.671594 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:32.674524 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:32.674529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:32.674529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:32.674542 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:32.676713 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:32.677138 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:32.677142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:32.677142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:32.677247 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:32.677477 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:32.677478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:32.677478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:32.677818 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:32.678018 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:32.678557 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 17:41:32.678730 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260924 17:41:32.704762 15793 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:41:32.705338 15793 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:32.708601 15793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Generated new TSK 0
I20260924 17:41:32.708786 15793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:41:32.737804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:32.819155 15810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:32.819355 15810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:32.819377 15810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:32.820436 15810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:32.820469 15810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:32.820495 15810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:32.821822 15810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:32.822046 15810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:32.822249 15810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:32.827327 15815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:32.827397 15816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.827471 15810 server_base.cc:1034] running on GCE node
W20260924 17:41:32.827327 15818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.827715 15810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:32.827966 15810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:32.829099 15810 hybrid_clock.cc:648] HybridClock initialized: now 1790271692829081 us; error 38 us; skew 500 ppm
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:32.831580 15810 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:32.832007 15810 webserver.cc:460] Webserver started at http://127.0.98.1:40675/ using document root <none> and password file <none>
I20260924 17:41:32.832253 15810 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:32.832298 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:32.832386 15810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:32.833385 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "250e1986eaef49048f7056fced5c5d9b"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "e59d6c51e807ce991bd2f101c9839a3b"
server_key_iv: "6bfdb8aacf7e51c1fc4777e320d83a17"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.833678 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "250e1986eaef49048f7056fced5c5d9b"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "e59d6c51e807ce991bd2f101c9839a3b"
server_key_iv: "6bfdb8aacf7e51c1fc4777e320d83a17"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.834913 15810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:32.835691 15825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.835880 15810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.835966 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "250e1986eaef49048f7056fced5c5d9b"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "e59d6c51e807ce991bd2f101c9839a3b"
server_key_iv: "6bfdb8aacf7e51c1fc4777e320d83a17"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.836041 15810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:32.851343 15810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:32.853313 15810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:32.853492 15810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:32.853742 15810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:32.854045 15810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:32.854079 15810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.854123 15810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:32.854149 15810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.859881 15810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35869
I20260924 17:41:32.859961 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35869 every 8 connection(s)
I20260924 17:41:32.860280 15810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:32.862623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15810
I20260924 17:41:32.862735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:41:32.862946 392 external_mini_cluster.cc:1428] Setting key cfb7467bc22de4b331f8db2be3a9b011
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:41:32.871766 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:32.861015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56939 (local address 127.0.98.62:34549)
0924 17:41:32.861082 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:32.861084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:32.861269 (+ 185us) server_negotiation.cc:407] Connection header received
0924 17:41:32.861805 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:32.861807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:32.861819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:32.861866 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:32.863046 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.863576 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.865817 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.866032 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.866089 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:32.868635 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:32.868637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:32.868637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:32.868646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:32.870257 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:32.870680 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:32.870682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:32.870682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:32.870747 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:32.870990 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:32.870991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:32.870991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:32.871169 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:32.871239 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:32.871553 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 17:41:32.871658 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:41:32.871847 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:32.861004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:56939)
0924 17:41:32.861157 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:32.861192 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:32.861693 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:32.861941 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:32.861944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:32.862582 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:32.862886 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:32.862893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.864826 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.864827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:32.865737 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:32.865741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:32.866118 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:32.866119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:32.866274 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:32.866787 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:32.866799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:32.868527 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:32.870353 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:32.870361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:32.870363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:32.870598 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:32.870823 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:32.870823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:32.870824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:32.870924 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:32.871235 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:32.871241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:32.871362 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 17:41:32.871520 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 17:41:32.871654 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260924 17:41:32.872331 15939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:32.872454 15939 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:32.872661 15939 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:32.873195 15747 ts_manager.cc:194] Registered new tserver with Master: 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869)
I20260924 17:41:32.873988 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:41:32.889864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:32.972865 15944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:32.973073 15944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:32.973095 15944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:32.974263 15944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:32.974305 15944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:32.974331 15944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:32.975829 15944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:32.976071 15944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:32.976322 15944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:32.981339 15952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:32.981470 15949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.981384 15944 server_base.cc:1034] running on GCE node
W20260924 17:41:32.981369 15950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:32.981714 15944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:32.981896 15944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:32.983100 15944 hybrid_clock.cc:648] HybridClock initialized: now 1790271692983092 us; error 26 us; skew 500 ppm
Sep 24 17:41:32 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:32.985457 15944 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:32.985905 15944 webserver.cc:460] Webserver started at http://127.0.98.2:34175/ using document root <none> and password file <none>
I20260924 17:41:32.986119 15944 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:32.986162 15944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:32.986320 15944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:32.987385 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb948705f88640e8b63a6db1f7656f3f"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "dda3d58a401475ec2b4ef3ff5e5ebbbb"
server_key_iv: "f56495509db96892bf438d7d0b261269"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.987730 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb948705f88640e8b63a6db1f7656f3f"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "dda3d58a401475ec2b4ef3ff5e5ebbbb"
server_key_iv: "f56495509db96892bf438d7d0b261269"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.988981 15944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:32.989640 15959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:32.989820 15944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:41:32.989903 15944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "cb948705f88640e8b63a6db1f7656f3f"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "dda3d58a401475ec2b4ef3ff5e5ebbbb"
server_key_iv: "f56495509db96892bf438d7d0b261269"
server_key_version: "encryptionkey@0"
I20260924 17:41:32.989972 15944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:32.999671 15944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:33.001567 15944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:33.001709 15944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:33.001948 15944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:33.002241 15944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:33.002274 15944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:33.002314 15944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:33.002341 15944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:33.008150 15944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38477
I20260924 17:41:33.008198 16072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38477 every 8 connection(s)
I20260924 17:41:33.008502 15944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:41:33 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:33.014966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 15944
I20260924 17:41:33.015092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:41:33.015303 392 external_mini_cluster.cc:1428] Setting key f789ffa06a3e5fc60164d9d574749191
I20260924 17:41:33.017706 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.009206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35739 (local address 127.0.98.62:34549)
0924 17:41:33.009272 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.009274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.009466 (+ 192us) server_negotiation.cc:407] Connection header received
0924 17:41:33.009987 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.009988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.010000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.010063 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:33.010981 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.011521 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.012517 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.012725 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.012810 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.014873 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:33.014875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:33.014875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:33.014883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:33.016367 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.016769 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.016770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.016770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.016825 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.017082 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.017082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.017083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.017195 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:33.017267 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.017539 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.017615 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:41:33.017905 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.009197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:35739)
0924 17:41:33.009344 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.009379 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.009875 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.010138 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.010141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.010689 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:33.010874 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.010879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.011634 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.011635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.012406 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.012408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.012818 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.012820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.012934 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.013452 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:33.013478 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:33.014777 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:33.016449 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:33.016455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:33.016464 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:33.016675 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:33.016898 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:33.016899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:33.016900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:33.017010 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:33.017259 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:33.017265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:33.017403 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.017518 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.017671 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 17:41:33.018482 16073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:33.018575 16073 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:33.018743 16073 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:33.019133 15747 ts_manager.cc:194] Registered new tserver with Master: cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:33.019516 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35739
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:41:33.041383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:33.122275 16078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:33.122481 16078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:33.122504 16078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:33.123623 16078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:33.123663 16078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:33.123689 16078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:33.125532 16078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:33.125741 16078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:33.125932 16078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:33.130934 16086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:33.131110 16078 server_base.cc:1034] running on GCE node
W20260924 17:41:33.130996 16083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:33.130937 16084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:33.131386 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:33.131592 16078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:33.132781 16078 hybrid_clock.cc:648] HybridClock initialized: now 1790271693132766 us; error 31 us; skew 500 ppm
Sep 24 17:41:33 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:33.135428 16078 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:33.135866 16078 webserver.cc:460] Webserver started at http://127.0.98.3:39513/ using document root <none> and password file <none>
I20260924 17:41:33.136054 16078 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:33.136096 16078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:33.136186 16078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:41:33.137233 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d806dd6da1e4a4693e02d805d53640c"
format_stamp: "Formatted at 2026-09-24 17:41:33 on dist-test-slave-d0pf"
server_key: "114aced41fb595eac539274eb600d338"
server_key_iv: "3756b7f15ca81b5d7ab8eb22ebf9d7bf"
server_key_version: "encryptionkey@0"
I20260924 17:41:33.137533 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d806dd6da1e4a4693e02d805d53640c"
format_stamp: "Formatted at 2026-09-24 17:41:33 on dist-test-slave-d0pf"
server_key: "114aced41fb595eac539274eb600d338"
server_key_iv: "3756b7f15ca81b5d7ab8eb22ebf9d7bf"
server_key_version: "encryptionkey@0"
I20260924 17:41:33.139364 16078 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 17:41:33.140097 16093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:33.140265 16078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:41:33.140340 16078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "9d806dd6da1e4a4693e02d805d53640c"
format_stamp: "Formatted at 2026-09-24 17:41:33 on dist-test-slave-d0pf"
server_key: "114aced41fb595eac539274eb600d338"
server_key_iv: "3756b7f15ca81b5d7ab8eb22ebf9d7bf"
server_key_version: "encryptionkey@0"
I20260924 17:41:33.140398 16078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:33.156139 16078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:33.158170 16078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:33.158286 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:33.158478 16078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:33.158818 16078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:33.158852 16078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:33.158877 16078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:33.158926 16078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:33.164729 16078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43953
I20260924 17:41:33.164808 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43953 every 8 connection(s)
I20260924 17:41:33.165130 16078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:33.166258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16078
I20260924 17:41:33.166371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:41:33.166595 392 external_mini_cluster.cc:1428] Setting key 3b60e4fe359fbfc0ef130d649c2af912
Sep 24 17:41:33 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:33.174362 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.165893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42255 (local address 127.0.98.62:34549)
0924 17:41:33.165975 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.165977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.166155 (+ 178us) server_negotiation.cc:407] Connection header received
0924 17:41:33.166792 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.166794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.166807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.166853 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:33.167890 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.168409 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.169409 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.169619 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.169706 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.171786 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:33.171788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:33.171788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:33.171796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:33.172965 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.173403 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.173404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.173405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.173467 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.173716 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.173717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.173717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.173836 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:33.173909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.174178 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.174258 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:41:33.174563 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.165856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:42255)
0924 17:41:33.166026 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.166069 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.166676 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.166937 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.166941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.167582 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:33.167782 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.167786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.168504 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.168506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.169316 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.169318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.169716 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.169717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.169857 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.170324 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:33.170331 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:33.171690 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:33.173062 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:33.173069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:33.173071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:33.173305 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:33.173536 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:33.173537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:33.173537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:33.173643 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:33.173914 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:33.173921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:33.174055 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.174159 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.174318 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260924 17:41:33.175096 16207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:33.175220 16207 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:33.175376 16207 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:33.175746 15747 ts_manager.cc:194] Registered new tserver with Master: 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953)
I20260924 17:41:33.176129 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42255
I20260924 17:41:33.178398 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:33.185353 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.179771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.98.62:34549)
0924 17:41:33.179833 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.179835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.179907 (+ 72us) server_negotiation.cc:407] Connection header received
0924 17:41:33.179948 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.179948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.179957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.180003 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:33.180512 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.180973 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.181678 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.181854 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.181901 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.182965 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:33.182966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:33.182967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:33.182972 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:33.184334 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.184695 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.184695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.184695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.184743 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:33.184900 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:33.184901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:33.184901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:33.185031 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:33.185103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.185196 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.185254 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:41:33.186928 15747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:41:33.188572 15747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:41:33.195215 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.190840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.0.98.3:43953)
0924 17:41:33.191054 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.191058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.191069 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:41:33.191107 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.191109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.191124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.191220 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:33.191469 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192605 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194257 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194744 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.194890 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.195074 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.195117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260924 17:41:33.195262 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.190730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43953 (local address 127.0.0.1:39188)
0924 17:41:33.190859 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.190903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.190973 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.191312 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.191315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.191317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:33.191389 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.191390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192710 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.192711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.194198 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.194200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194840 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.194927 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.194940 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.195033 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.195126 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":58,"thread_start_us":57,"threads_started":1}
I20260924 17:41:33.195377 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.191083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.0.98.1:35869)
0924 17:41:33.191272 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.191276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.191431 (+ 155us) server_negotiation.cc:407] Connection header received
0924 17:41:33.191478 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.191481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.191496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.191567 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:33.191792 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192821 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194038 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194773 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194847 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.194906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.195249 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.195307 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260924 17:41:33.195441 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.191054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.0.98.2:38477)
0924 17:41:33.191239 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.191244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.191445 (+ 201us) server_negotiation.cc:407] Connection header received
0924 17:41:33.191485 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.191487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.191501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.191567 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:33.191785 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192620 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194193 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194862 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194920 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.194966 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.195299 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.195340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260924 17:41:33.195582 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.190730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.0.1:36934)
0924 17:41:33.191238 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.191309 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.191368 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.191643 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.191644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.191649 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:33.191700 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.191702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192928 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.192930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.193954 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.193957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194942 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.195009 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.195012 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.195082 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.195136 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":6,"thread_start_us":65,"threads_started":1}
I20260924 17:41:33.195266 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.190918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.0.1:51464)
0924 17:41:33.191321 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.191332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.191386 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.191643 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.191644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.191646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:33.191710 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.191711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.192734 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.192734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.194111 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.194114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.194994 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.194995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.195085 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.195088 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.195132 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.195168 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":93,"thread_start_us":50,"threads_started":1}
I20260924 17:41:33.196300 16141 tablet_service.cc:1467] Processing CreateTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f (DEFAULT_TABLE table=test-table [id=1ca48f97e9e04a4b94203e7b8d764632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:33.196393 15873 tablet_service.cc:1467] Processing CreateTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f (DEFAULT_TABLE table=test-table [id=1ca48f97e9e04a4b94203e7b8d764632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:33.196394 16007 tablet_service.cc:1467] Processing CreateTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f (DEFAULT_TABLE table=test-table [id=1ca48f97e9e04a4b94203e7b8d764632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:33.196646 16141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f5d64e752f41c58a7166fee3bf6f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:33.196713 16007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f5d64e752f41c58a7166fee3bf6f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:33.196713 15873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f5d64e752f41c58a7166fee3bf6f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:33.199095 16221 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Bootstrap starting.
I20260924 17:41:33.199646 16221 tablet_bootstrap.cc:654] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:33.200155 16222 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Bootstrap starting.
I20260924 17:41:33.200268 16221 log.cc:824] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:33.200834 16222 tablet_bootstrap.cc:654] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:33.201040 16221 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: No bootstrap required, opened a new log
I20260924 17:41:33.201115 16221 ts_tablet_manager.cc:1397] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:33.201367 16222 log.cc:824] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:33.201694 16223 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Bootstrap starting.
I20260924 17:41:33.202112 16222 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: No bootstrap required, opened a new log
I20260924 17:41:33.202220 16222 ts_tablet_manager.cc:1397] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:33.202268 16223 tablet_bootstrap.cc:654] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:33.202718 16223 log.cc:824] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:33.202948 16221 raft_consensus.cc:348] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.203156 16221 raft_consensus.cc:374] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:33.203213 16221 raft_consensus.cc:729] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d806dd6da1e4a4693e02d805d53640c, State: Initialized, Role: FOLLOWER
I20260924 17:41:33.203378 16221 consensus_queue.cc:260] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.203554 16223 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: No bootstrap required, opened a new log
I20260924 17:41:33.203610 16223 ts_tablet_manager.cc:1397] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:41:33.203644 16221 ts_tablet_manager.cc:1428] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:41:33.203753 16207 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260924 17:41:33.204228 16222 raft_consensus.cc:348] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.204350 16222 raft_consensus.cc:374] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:33.204373 16222 raft_consensus.cc:729] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e1986eaef49048f7056fced5c5d9b, State: Initialized, Role: FOLLOWER
I20260924 17:41:33.204444 16222 consensus_queue.cc:260] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.204674 16222 ts_tablet_manager.cc:1428] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:33.204864 15939 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260924 17:41:33.205557 16223 raft_consensus.cc:348] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.205683 16223 raft_consensus.cc:374] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:33.205703 16223 raft_consensus.cc:729] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb948705f88640e8b63a6db1f7656f3f, State: Initialized, Role: FOLLOWER
I20260924 17:41:33.205808 16223 consensus_queue.cc:260] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.206025 16223 ts_tablet_manager.cc:1428] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:41:33.206086 16073 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260924 17:41:33.228688 16227 raft_consensus.cc:482] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:33.228816 16227 raft_consensus.cc:504] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.229259 16227 leader_election.cc:288] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:33.233414 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.229799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45205 (local address 127.0.98.2:38477)
0924 17:41:33.229851 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.229853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.229863 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:33.229889 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.229890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.229902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.229959 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:33.230097 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.230843 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.232409 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.233016 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.233095 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.233106 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.233299 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.233340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:41:33.233414 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.229537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43045 (local address 127.0.98.1:35869)
0924 17:41:33.229599 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.229601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.229776 (+ 175us) server_negotiation.cc:407] Connection header received
0924 17:41:33.229807 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.229808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.229822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.229871 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:33.230088 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.230876 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.232416 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.233014 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.233095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.233106 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.233299 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.233342 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:41:33.233429 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.229458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.98.3:43045)
0924 17:41:33.229532 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.229544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.229725 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.229895 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.229895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.229897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:33.229976 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.229978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.230950 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.230951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.232281 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.232284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.233104 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.233105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.233206 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.233213 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.233275 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.233338 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":51}
I20260924 17:41:33.233438 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.229570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.98.3:45205)
0924 17:41:33.229753 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.229765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:33.229827 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:33.229965 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:33.229966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:33.229967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:33.230025 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.230026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.230944 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.230945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.232281 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:33.232284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.233104 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.233105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:33.233206 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.233213 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 17:41:33.233275 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:41:33.233329 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":77,"thread_start_us":135,"threads_started":1}
I20260924 17:41:33.233804 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f5d64e752f41c58a7166fee3bf6f4f" candidate_uuid: "9d806dd6da1e4a4693e02d805d53640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250e1986eaef49048f7056fced5c5d9b" is_pre_election: true
I20260924 17:41:33.233826 16027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f5d64e752f41c58a7166fee3bf6f4f" candidate_uuid: "9d806dd6da1e4a4693e02d805d53640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb948705f88640e8b63a6db1f7656f3f" is_pre_election: true
I20260924 17:41:33.233983 16027 raft_consensus.cc:2454] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d806dd6da1e4a4693e02d805d53640c in term 0.
I20260924 17:41:33.233983 15893 raft_consensus.cc:2454] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d806dd6da1e4a4693e02d805d53640c in term 0.
I20260924 17:41:33.234221 16094 leader_election.cc:302] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b, 9d806dd6da1e4a4693e02d805d53640c; no voters:
I20260924 17:41:33.234355 16227 raft_consensus.cc:2781] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:33.234409 16227 raft_consensus.cc:482] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:33.234438 16227 raft_consensus.cc:3037] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:33.235190 16227 raft_consensus.cc:504] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.235337 16227 leader_election.cc:288] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [CANDIDATE]: Term 1 election: Requested vote from peers 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:33.235529 16027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f5d64e752f41c58a7166fee3bf6f4f" candidate_uuid: "9d806dd6da1e4a4693e02d805d53640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb948705f88640e8b63a6db1f7656f3f"
I20260924 17:41:33.235564 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f5d64e752f41c58a7166fee3bf6f4f" candidate_uuid: "9d806dd6da1e4a4693e02d805d53640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250e1986eaef49048f7056fced5c5d9b"
I20260924 17:41:33.235635 15893 raft_consensus.cc:3037] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:33.235637 16027 raft_consensus.cc:3037] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:33.236373 15893 raft_consensus.cc:2454] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d806dd6da1e4a4693e02d805d53640c in term 1.
I20260924 17:41:33.236393 16027 raft_consensus.cc:2454] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d806dd6da1e4a4693e02d805d53640c in term 1.
I20260924 17:41:33.236543 16094 leader_election.cc:302] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d806dd6da1e4a4693e02d805d53640c, cb948705f88640e8b63a6db1f7656f3f; no voters:
I20260924 17:41:33.236694 16227 raft_consensus.cc:2781] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:33.236853 16227 raft_consensus.cc:686] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 1 LEADER]: Becoming Leader. State: Replica: 9d806dd6da1e4a4693e02d805d53640c, State: Running, Role: LEADER
I20260924 17:41:33.236958 16227 consensus_queue.cc:237] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:33.237800 15747 catalog_manager.cc:5495] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9d806dd6da1e4a4693e02d805d53640c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:41:33.253752 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.251464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.3:43953)
0924 17:41:33.251532 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:33.251535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:33.251548 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:41:33.251579 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:33.251580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:33.251597 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:33.251657 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:33.251768 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.252225 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.252960 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:33.253169 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:33.253234 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:33.253380 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:33.253478 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:33.253539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:33.253575 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 17:41:33.253593 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:41:33.255510 16027 raft_consensus.cc:1261] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Refusing update from remote peer 9d806dd6da1e4a4693e02d805d53640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:33.255532 15893 raft_consensus.cc:1261] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Refusing update from remote peer 9d806dd6da1e4a4693e02d805d53640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:33.255905 16231 consensus_queue.cc:1035] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:33.256045 16227 consensus_queue.cc:1035] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:33.258205 16234 mvcc.cc:205] Tried to move back new op lower bound from 7332952855572082688 to 7332952855499739136. Current Snapshot: MvccSnapshot[applied={T|T < 7332952855572082688}]
I20260924 17:41:33.258505 16236 mvcc.cc:205] Tried to move back new op lower bound from 7332952855572082688 to 7332952855499739136. Current Snapshot: MvccSnapshot[applied={T|T < 7332952855572082688}]
I20260924 17:41:33.258867 16235 mvcc.cc:205] Tried to move back new op lower bound from 7332952855572082688 to 7332952855499739136. Current Snapshot: MvccSnapshot[applied={T|T < 7332952855572082688}]
I20260924 17:41:33.261171 15747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:33.261269 15747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:33.261996 15747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Sending DeleteTablet for 3 replicas of tablet 73f5d64e752f41c58a7166fee3bf6f4f
I20260924 17:41:33.262342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15810
I20260924 17:41:33.262399 16141 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39188
I20260924 17:41:33.262403 16007 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51464
I20260924 17:41:33.263288 16245 tablet_replica.cc:330] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: stopping tablet replica
I20260924 17:41:33.263391 16245 raft_consensus.cc:2229] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:41:33.263510 16245 raft_consensus.cc:2258] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:33.263954 16245 ts_tablet_manager.cc:1905] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:33.265969 16245 ts_tablet_manager.cc:1918] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:33.266031 16245 log.cc:1196] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/73f5d64e752f41c58a7166fee3bf6f4f
I20260924 17:41:33.266304 16245 ts_tablet_manager.cc:1939] T 73f5d64e752f41c58a7166fee3bf6f4f P 9d806dd6da1e4a4693e02d805d53640c: Deleting consensus metadata
I20260924 17:41:33.266784 15733 catalog_manager.cc:4841] TS 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): tablet 73f5d64e752f41c58a7166fee3bf6f4f (table test-table [id=1ca48f97e9e04a4b94203e7b8d764632]) successfully deleted
I20260924 17:41:33.267445 16244 tablet_replica.cc:330] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: stopping tablet replica
I20260924 17:41:33.267592 16244 raft_consensus.cc:2229] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:33.267715 16244 raft_consensus.cc:2258] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:33.268172 16244 ts_tablet_manager.cc:1905] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:41:33.268776 15732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35869 (error 108)
I20260924 17:41:33.269011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15944
I20260924 17:41:33.269809 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.269683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.0.1:36936)
0924 17:41:33.269739 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.269777 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:41:33.269950 15732 catalog_manager.cc:4584] TS 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73f5d64e752f41c58a7166fee3bf6f4f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35869: connect: Connection refused (error 111)
I20260924 17:41:33.275208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16078
I20260924 17:41:33.275228 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:33.275114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.0.1:51480)
0924 17:41:33.275166 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:33.275193 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:41:33.280171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 15716
I20260924 17:41:53.287643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--webserver_interface=127.0.98.62
--webserver_port=43401
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:41:53.372197 16247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:53.372438 16247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:53.372463 16247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:53.373735 16247 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:41:53.373788 16247 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:41:53.373803 16247 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:41:53.373816 16247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:41:53.373829 16247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:53.373843 16247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:53.373855 16247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:41:53.373914 16247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:41:53.375581 16247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:53.375874 16247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:53.376144 16247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:53.385232 16253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.385251 16254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.385317 16256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:53.385294 16247 server_base.cc:1034] running on GCE node
I20260924 17:41:53.386020 16247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:53.386341 16247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:53.387531 16247 hybrid_clock.cc:648] HybridClock initialized: now 1790271713387511 us; error 43 us; skew 500 ppm
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:53.397094 16247 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:41:53.397666 16247 webserver.cc:460] Webserver started at http://127.0.98.62:43401/ using document root <none> and password file <none>
I20260924 17:41:53.397945 16247 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:53.398034 16247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:53.399703 16247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:53.400509 16263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.400727 16247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:41:53.400842 16247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f7c320746dab4013832325a3a6c1bf50"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "16b8d28f38c0ed166cc0e4ba19da5f5f"
server_key_iv: "a961ac17fdc00b1cf0b2a4eaa3f3e12d"
server_key_version: "encryptionkey@0"
I20260924 17:41:53.401283 16247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:53.415845 16247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:53.418131 16247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:53.418310 16247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:53.422655 16247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34549
I20260924 17:41:53.422695 16315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34549 every 8 connection(s)
I20260924 17:41:53.423143 16247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:41:53.424638 16316 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:41:53.425333 16316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap starting.
I20260924 17:41:53.433363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16247
I20260924 17:41:53.433709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35869
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=40675
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:53.433933 16316 log.cc:824] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:53.436692 16316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:53.437026 16316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap complete.
I20260924 17:41:53.439544 16316 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:53.439795 16316 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.439975 16316 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:53.440052 16316 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:41:53.440110 16316 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:41:53.440163 16316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:41:53.441255 16316 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:53.441354 16316 leader_election.cc:302] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c320746dab4013832325a3a6c1bf50; no voters:
I20260924 17:41:53.441704 16316 leader_election.cc:288] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:41:53.441745 16320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:41:53.441871 16320 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 LEADER]: Becoming Leader. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Running, Role: LEADER
I20260924 17:41:53.441998 16316 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:41:53.441972 16320 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:41:53.442561 16322 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:41:53.442680 16322 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:53.443133 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c320746dab4013832325a3a6c1bf50. Latest consensus state: current_term: 2 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:41:53.443230 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:41:53.443409 16329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:41:53.444411 16329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:41:53.444571 16329 catalog_manager.cc:1259] Loaded cluster ID: 719b21d9978b43a695602a0dc7367824
I20260924 17:41:53.444805 16329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:41:53.446156 16329 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:41:53.446355 16329 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Loaded TSK: 0
I20260924 17:41:53.456211 16329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Generated new TSK 1
I20260924 17:41:53.456326 16329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:41:53.526037 16319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:53.526316 16319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:53.526362 16319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:53.527719 16319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:53.527791 16319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:53.527839 16319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:41:53.529623 16319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40675
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:53.529922 16319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:53.530170 16319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:53.536440 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.536455 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.536551 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:53.536722 16319 server_base.cc:1034] running on GCE node
I20260924 17:41:53.536953 16319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:53.537295 16319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:53.538489 16319 hybrid_clock.cc:648] HybridClock initialized: now 1790271713538476 us; error 27 us; skew 500 ppm
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:53.541386 16319 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:41:53.541975 16319 webserver.cc:460] Webserver started at http://127.0.98.1:40675/ using document root <none> and password file <none>
I20260924 17:41:53.542233 16319 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:53.542326 16319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:53.543974 16319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:53.544713 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.544912 16319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:53.545001 16319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "250e1986eaef49048f7056fced5c5d9b"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "e59d6c51e807ce991bd2f101c9839a3b"
server_key_iv: "6bfdb8aacf7e51c1fc4777e320d83a17"
server_key_version: "encryptionkey@0"
I20260924 17:41:53.545475 16319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:53.564788 16319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:53.568041 16319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:53.568215 16319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:53.568493 16319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:53.569016 16360 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:41:53.570601 16319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:41:53.570662 16319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:41:53.570703 16319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:41:53.571568 16319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:41:53.571610 16319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:53.571700 16360 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Bootstrap starting.
I20260924 17:41:53.578248 16319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35869
I20260924 17:41:53.578544 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35869 every 8 connection(s)
I20260924 17:41:53.578740 16319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:41:53.583500 16360 log.cc:824] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:53.583544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16319
I20260924 17:41:53.583846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38477
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=34175
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:41:53.586285 16360 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:53.586630 16360 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Bootstrap complete.
I20260924 17:41:53.587906 16360 ts_tablet_manager.cc:1397] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:53.589651 16360 raft_consensus.cc:348] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.590081 16360 raft_consensus.cc:729] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e1986eaef49048f7056fced5c5d9b, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.590325 16360 consensus_queue.cc:260] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.590648 16360 ts_tablet_manager.cc:1428] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:41:53.593053 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.579502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:52949)
0924 17:41:53.579739 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.579789 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.580457 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.581002 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.581007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.581356 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:53.581612 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.581617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.583479 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.583480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.585079 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.585082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.586040 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.586042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.586185 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.586895 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:53.586906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:53.588797 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:53.591327 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.591334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.591336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.591573 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.591869 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.591870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.591871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.591992 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.592398 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:53.592404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:53.592549 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.592669 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.592857 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260924 17:41:53.593055 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.579570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52949 (local address 127.0.98.62:34549)
0924 17:41:53.580476 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.580486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.580534 (+ 48us) server_negotiation.cc:407] Connection header received
0924 17:41:53.580700 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.580708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.580784 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.580962 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:53.581765 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.583316 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.585194 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.585805 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.585927 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.588931 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:53.588936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:53.588936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:53.588950 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:53.591220 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.591682 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.591686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.591687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.591796 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.592065 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.592066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.592067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.592320 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:53.592419 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.592722 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.592855 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":76,"threads_started":1}
I20260924 17:41:53.593823 16468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:53.593914 16468 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:53.594089 16468 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:53.594830 16280 ts_manager.cc:194] Registered new tserver with Master: 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869)
I20260924 17:41:53.595999 16280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52949
I20260924 17:41:53.596848 16468 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260924 17:41:53.600754 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.595475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.0.1:53188)
0924 17:41:53.597008 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.597049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.597124 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.597428 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.597431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.597433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:53.597486 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.597487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.598732 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.598733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.599628 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.599631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.600528 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.600529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.600590 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.600601 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.600652 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.600693 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"thread_start_us":95,"threads_started":1}
I20260924 17:41:53.600782 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.595725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.1:35869)
0924 17:41:53.597267 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.597272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.597281 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:41:53.597314 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.597316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.597330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.597399 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:53.597555 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.598630 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.599711 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.600443 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.600524 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.600580 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.600673 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.600711 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"thread_start_us":66,"threads_started":1}
I20260924 17:41:53.601334 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260924 17:41:53.601866 16477 tablet_replica.cc:330] stopping tablet replica
I20260924 17:41:53.601974 16477 raft_consensus.cc:2229] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:53.602044 16477 raft_consensus.cc:2258] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:53.602375 16477 ts_tablet_manager.cc:1905] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:53.604667 16477 ts_tablet_manager.cc:1918] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:53.604732 16477 log.cc:1196] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal/wals/73f5d64e752f41c58a7166fee3bf6f4f
I20260924 17:41:53.604980 16477 ts_tablet_manager.cc:1939] T 73f5d64e752f41c58a7166fee3bf6f4f P 250e1986eaef49048f7056fced5c5d9b: Deleting consensus metadata
I20260924 17:41:53.605547 16264 catalog_manager.cc:4841] TS 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869): tablet 73f5d64e752f41c58a7166fee3bf6f4f (table test-table [id=1ca48f97e9e04a4b94203e7b8d764632]) successfully deleted
W20260924 17:41:53.674520 16473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:53.674801 16473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:53.674844 16473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:53.676142 16473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:53.676220 16473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:53.676268 16473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:41:53.677810 16473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34175
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:53.678092 16473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:53.678345 16473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:53.683691 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.683706 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.683700 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:53.684013 16473 server_base.cc:1034] running on GCE node
I20260924 17:41:53.684270 16473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:53.684547 16473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:53.685731 16473 hybrid_clock.cc:648] HybridClock initialized: now 1790271713685724 us; error 55 us; skew 500 ppm
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:53.688470 16473 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:41:53.688975 16473 webserver.cc:460] Webserver started at http://127.0.98.2:34175/ using document root <none> and password file <none>
I20260924 17:41:53.689229 16473 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:53.689320 16473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:53.690866 16473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:41:53.691768 16492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.691982 16473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:41:53.692098 16473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "cb948705f88640e8b63a6db1f7656f3f"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "dda3d58a401475ec2b4ef3ff5e5ebbbb"
server_key_iv: "f56495509db96892bf438d7d0b261269"
server_key_version: "encryptionkey@0"
I20260924 17:41:53.692497 16473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:53.714637 16473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:53.717198 16473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:53.717386 16473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:53.717645 16473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:53.718080 16499 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:41:53.719581 16473 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:41:53.719645 16473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:41:53.719705 16473 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:41:53.720383 16473 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:41:53.720432 16473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:53.720556 16499 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Bootstrap starting.
I20260924 17:41:53.722162 16499 log.cc:824] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:53.724675 16499 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:41:53.724988 16499 tablet_bootstrap.cc:492] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Bootstrap complete.
I20260924 17:41:53.725163 16499 ts_tablet_manager.cc:1397] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 17:41:53.726636 16499 raft_consensus.cc:348] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.726924 16499 raft_consensus.cc:729] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb948705f88640e8b63a6db1f7656f3f, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.727144 16499 consensus_queue.cc:260] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.727455 16499 ts_tablet_manager.cc:1428] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260924 17:41:53.727831 16473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38477
I20260924 17:41:53.727869 16608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38477 every 8 connection(s)
I20260924 17:41:53.728190 16473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:41:53.730646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16473
I20260924 17:41:53.730932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43953
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=39513
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:53.738869 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.728932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:41999)
0924 17:41:53.729109 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.729152 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.729335 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.729582 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.729586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.729858 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:53.730059 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.730064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.730874 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.730875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.732225 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.732229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.732811 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.732812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.732916 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.733493 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:53.733501 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:53.735353 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:53.737407 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.737416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.737418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.737678 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.737906 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.737907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.737907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.738006 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.738333 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:53.738339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:53.738466 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.738563 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.738705 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260924 17:41:53.738863 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.728963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41999 (local address 127.0.98.62:34549)
0924 17:41:53.729051 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.729053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.729290 (+ 237us) server_negotiation.cc:407] Connection header received
0924 17:41:53.729420 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.729421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.729437 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.729499 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:53.730170 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.730681 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.732343 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.732708 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.732790 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.735462 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:53.735464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:53.735464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:53.735473 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:53.737302 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.737756 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.737758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.737759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.737834 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.738073 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.738074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.738074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.738236 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:53.738319 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.738609 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.738718 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:41:53.739547 16609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:53.739666 16609 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:53.739895 16609 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:53.740581 16280 ts_manager.cc:194] Registered new tserver with Master: cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:53.741255 16280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41999
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 17:41:53.749944 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.740906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.0.1:60866)
0924 17:41:53.740959 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.740970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.741011 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.741284 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.741285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.741684 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:53.741731 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.741734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.742983 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.742984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.743825 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.743828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.744343 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.744347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.744432 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.744990 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:53.744995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:53.746607 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:53.748710 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.748718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.748721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.748932 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.749171 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.749171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.749172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.749234 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.749535 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:53.749543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:53.749606 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.749694 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.749850 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:41:53.749956 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.740973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.0.98.2:38477)
0924 17:41:53.741130 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.741135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.741151 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:41:53.741180 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.741181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.741192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.741270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:53.742127 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.742890 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.743929 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.744252 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.744342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.746730 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:53.746741 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:53.746741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:53.746753 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:53.748470 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.749023 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.749026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.749027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.749092 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.749299 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.749300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.749300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.749460 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:53.749556 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.749729 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.749839 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260924 17:41:53.750335 16542 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f5d64e752f41c58a7166fee3bf6f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60866
I20260924 17:41:53.750912 16614 tablet_replica.cc:330] stopping tablet replica
I20260924 17:41:53.751035 16614 raft_consensus.cc:2229] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:53.751137 16614 raft_consensus.cc:2258] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:53.751451 16614 ts_tablet_manager.cc:1905] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:53.753854 16614 ts_tablet_manager.cc:1918] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:53.753937 16614 log.cc:1196] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/73f5d64e752f41c58a7166fee3bf6f4f
I20260924 17:41:53.754199 16614 ts_tablet_manager.cc:1939] T 73f5d64e752f41c58a7166fee3bf6f4f P cb948705f88640e8b63a6db1f7656f3f: Deleting consensus metadata
I20260924 17:41:53.754729 16264 catalog_manager.cc:4841] TS cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): tablet 73f5d64e752f41c58a7166fee3bf6f4f (table test-table [id=1ca48f97e9e04a4b94203e7b8d764632]) successfully deleted
W20260924 17:41:53.817901 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:41:53.818172 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:41:53.818215 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:41:53.819442 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:41:53.819518 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:41:53.819564 16612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:41:53.821044 16612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39513
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:41:53.821329 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:41:53.821579 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:41:53.826622 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.826639 16622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:41:53.826686 16620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:41:53.826964 16612 server_base.cc:1034] running on GCE node
I20260924 17:41:53.827407 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:41:53.827746 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:41:53.828920 16612 hybrid_clock.cc:648] HybridClock initialized: now 1790271713828916 us; error 38 us; skew 500 ppm
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:41:53.831615 16612 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:41:53.832057 16612 webserver.cc:460] Webserver started at http://127.0.98.3:39513/ using document root <none> and password file <none>
I20260924 17:41:53.832322 16612 fs_manager.cc:359] Metadata directory not provided
I20260924 17:41:53.832409 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:41:53.833722 16612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 17:41:53.834261 16629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.834424 16612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.834532 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "9d806dd6da1e4a4693e02d805d53640c"
format_stamp: "Formatted at 2026-09-24 17:41:33 on dist-test-slave-d0pf"
server_key: "114aced41fb595eac539274eb600d338"
server_key_iv: "3756b7f15ca81b5d7ab8eb22ebf9d7bf"
server_key_version: "encryptionkey@0"
I20260924 17:41:53.834892 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:41:53.843233 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:41:53.845515 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:41:53.845691 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:41:53.845961 16612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:41:53.846312 16612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:41:53.846371 16612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.846413 16612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:41:53.846439 16612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.852595 16612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43953
I20260924 17:41:53.852635 16742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43953 every 8 connection(s)
I20260924 17:41:53.852995 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:41:53.856213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16612
Sep 24 17:41:53 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:41:53.863971 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.856696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.0.98.62:34549)
0924 17:41:53.856884 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.856886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.856896 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:41:53.856962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.856963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.856975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.857074 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:53.857658 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.858344 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.859338 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.859618 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.859673 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.861001 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:53.861002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:53.861002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:53.861008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:53.862741 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.863151 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.863152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.863152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.863234 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.863443 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.863443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.863444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.863555 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:53.863739 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.863778 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.863865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":164,"thread_start_us":112,"threads_started":1}
I20260924 17:41:53.864118 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.853699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44025 (local address 127.0.98.62:34549)
0924 17:41:53.853775 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.853777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.853997 (+ 220us) server_negotiation.cc:407] Connection header received
0924 17:41:53.854538 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.854539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.854554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.854606 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:53.855638 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.856240 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.857742 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.857926 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.857977 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.860841 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:53.860843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:53.860843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:53.860851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:53.862536 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.863053 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.863054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.863055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.863113 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:53.863415 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:53.863416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:53.863416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:53.863540 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:53.863590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.863953 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.864035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:41:53.864380 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.853701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:44025)
0924 17:41:53.853873 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.853909 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.854420 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.854678 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.854682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.855325 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:41:53.855527 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.855532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.856366 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.856367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.857648 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.857651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.858008 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.858009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.858168 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.858804 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:41:53.858812 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:41:53.860747 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:41:53.862639 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.862648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.862651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.862946 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.863203 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:41:53.863204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:41:53.863204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:41:53.863347 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:41:53.863606 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:41:53.863625 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:41:53.863792 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.863909 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.864088 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260924 17:41:53.864761 16743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260924 17:41:53.864902 16743 heartbeater.cc:464] Registering TS with master...
I20260924 17:41:53.865104 16743 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260924 17:41:53.865502 16280 ts_manager.cc:194] Registered new tserver with Master: 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953)
I20260924 17:41:53.865900 16280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44025
I20260924 17:41:53.876216 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:41:53.879500 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.876689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.0.98.62:34549)
0924 17:41:53.876757 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.876759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.876940 (+ 181us) server_negotiation.cc:407] Connection header received
0924 17:41:53.876984 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.876985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.877030 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.877090 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:53.877289 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.877793 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.878628 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.878847 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.878929 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.879185 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:53.879288 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:53.879354 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.879399 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.879417 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:41:53.879856 16280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:41:53.881654 16280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:41:53.884068 16542 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea (DEFAULT_TABLE table=test-table [id=7bff420118c54e67826fd2813d4667b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:53.884291 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c55007d2f4edc8b3f158b8db040ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:53.885927 16499 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Bootstrap starting.
I20260924 17:41:53.886099 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea (DEFAULT_TABLE table=test-table [id=7bff420118c54e67826fd2813d4667b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:53.886269 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c55007d2f4edc8b3f158b8db040ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:53.886368 16499 tablet_bootstrap.cc:654] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:53.887708 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.883759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43953 (local address 127.0.0.1:60468)
0924 17:41:53.883813 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:53.883824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:53.883865 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:53.884325 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:53.884326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:53.884328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:53.884387 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.884389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.885617 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.885618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.886599 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:53.886601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.887466 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.887467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:53.887531 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.887536 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:41:53.887588 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:41:53.887634 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:41:53.887722 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:53.883931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.98.3:43953)
0924 17:41:53.884142 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:53.884147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:53.884159 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:41:53.884200 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:53.884202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:53.884234 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:53.884321 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:53.884461 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.885528 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.886685 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:53.887375 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:53.887466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:53.887528 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:53.887601 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 17:41:53.887640 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260924 17:41:53.887719 16360 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Bootstrap starting.
I20260924 17:41:53.888019 16360 tablet_bootstrap.cc:654] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:53.888422 16499 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: No bootstrap required, opened a new log
I20260924 17:41:53.888475 16499 ts_tablet_manager.cc:1397] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:41:53.888623 16499 raft_consensus.cc:348] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.888700 16499 raft_consensus.cc:374] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:53.888746 16499 raft_consensus.cc:729] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb948705f88640e8b63a6db1f7656f3f, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.888804 16499 consensus_queue.cc:260] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.888914 16499 ts_tablet_manager.cc:1428] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.888795 16677 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea (DEFAULT_TABLE table=test-table [id=7bff420118c54e67826fd2813d4667b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:41:53.888955 16609 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260924 17:41:53.889114 16360 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: No bootstrap required, opened a new log
I20260924 17:41:53.889130 16677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c55007d2f4edc8b3f158b8db040ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:41:53.889161 16360 ts_tablet_manager.cc:1397] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 17:41:53.889309 16360 raft_consensus.cc:348] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.889401 16360 raft_consensus.cc:374] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:53.889452 16360 raft_consensus.cc:729] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e1986eaef49048f7056fced5c5d9b, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.889504 16360 consensus_queue.cc:260] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.889605 16360 ts_tablet_manager.cc:1428] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:41:53.892100 16752 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Bootstrap starting.
I20260924 17:41:53.892783 16752 tablet_bootstrap.cc:654] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Neither blocks nor log segments found. Creating new log.
I20260924 17:41:53.893340 16752 log.cc:824] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:41:53.894025 16752 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: No bootstrap required, opened a new log
I20260924 17:41:53.894094 16752 ts_tablet_manager.cc:1397] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:53.895603 16752 raft_consensus.cc:348] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.895753 16752 raft_consensus.cc:374] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:41:53.895776 16752 raft_consensus.cc:729] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d806dd6da1e4a4693e02d805d53640c, State: Initialized, Role: FOLLOWER
I20260924 17:41:53.895892 16752 consensus_queue.cc:260] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:53.896200 16752 ts_tablet_manager.cc:1428] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:41:53.896198 16743 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
W20260924 17:41:53.978855 16610 tablet.cc:2369] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:54.039695 16474 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:41:54.039873 16474 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:54.040256 16474 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:54.044265 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.040472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43953 (local address 127.0.98.1:59241)
0924 17:41:54.040571 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:54.040584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:54.040664 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:54.040841 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:54.040841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:54.040843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:54.040932 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:54.040934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.041892 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.041893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:54.043244 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:54.043248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.043993 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.043994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:54.044050 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.044054 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:41:54.044120 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:41:54.044179 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":28}
I20260924 17:41:54.044291 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.040621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59241 (local address 127.0.98.3:43953)
0924 17:41:54.040679 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:54.040681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:54.040700 (+ 19us) server_negotiation.cc:407] Connection header received
0924 17:41:54.040727 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:54.040727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:54.040758 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:54.040823 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:54.041013 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.041809 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.043318 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.043913 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.043990 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.044001 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:54.044166 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 17:41:54.044210 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:41:54.044448 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.040472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.98.1:57735)
0924 17:41:54.040686 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:54.040696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:41:54.040749 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:41:54.040920 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:41:54.040920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:41:54.040922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:41:54.040991 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:54.040993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.041892 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.041893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:54.043318 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:41:54.043320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.044187 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.044188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:41:54.044250 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.044253 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:41:54.044296 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 17:41:54.044333 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":119,"thread_start_us":106,"threads_started":1}
I20260924 17:41:54.044457 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.040561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57735 (local address 127.0.98.2:38477)
0924 17:41:54.040608 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:54.040610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:54.040769 (+ 159us) server_negotiation.cc:407] Connection header received
0924 17:41:54.040813 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:54.040815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:54.040839 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:54.040909 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:41:54.041083 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.041806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.043397 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.044047 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.044132 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.044249 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:54.044319 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 17:41:54.044370 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:41:54.044658 16697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c55007d2f4edc8b3f158b8db040ea" candidate_uuid: "250e1986eaef49048f7056fced5c5d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d806dd6da1e4a4693e02d805d53640c" is_pre_election: true
I20260924 17:41:54.044777 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c55007d2f4edc8b3f158b8db040ea" candidate_uuid: "250e1986eaef49048f7056fced5c5d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb948705f88640e8b63a6db1f7656f3f" is_pre_election: true
I20260924 17:41:54.044823 16697 raft_consensus.cc:2454] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250e1986eaef49048f7056fced5c5d9b in term 0.
I20260924 17:41:54.044903 16562 raft_consensus.cc:2454] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250e1986eaef49048f7056fced5c5d9b in term 0.
I20260924 17:41:54.045059 16355 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b, 9d806dd6da1e4a4693e02d805d53640c; no voters:
I20260924 17:41:54.045208 16474 raft_consensus.cc:2781] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:41:54.045257 16474 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:41:54.045275 16474 raft_consensus.cc:3037] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:54.045943 16474 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:54.046094 16474 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 1 election: Requested vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
I20260924 17:41:54.046240 16697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c55007d2f4edc8b3f158b8db040ea" candidate_uuid: "250e1986eaef49048f7056fced5c5d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d806dd6da1e4a4693e02d805d53640c"
I20260924 17:41:54.046240 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c55007d2f4edc8b3f158b8db040ea" candidate_uuid: "250e1986eaef49048f7056fced5c5d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb948705f88640e8b63a6db1f7656f3f"
I20260924 17:41:54.046312 16562 raft_consensus.cc:3037] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:54.046314 16697 raft_consensus.cc:3037] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:41:54.047156 16697 raft_consensus.cc:2454] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250e1986eaef49048f7056fced5c5d9b in term 1.
I20260924 17:41:54.047156 16562 raft_consensus.cc:2454] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250e1986eaef49048f7056fced5c5d9b in term 1.
I20260924 17:41:54.047331 16354 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b, cb948705f88640e8b63a6db1f7656f3f; no voters:
I20260924 17:41:54.047420 16474 raft_consensus.cc:2781] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:41:54.047537 16474 raft_consensus.cc:686] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 LEADER]: Becoming Leader. State: Replica: 250e1986eaef49048f7056fced5c5d9b, State: Running, Role: LEADER
I20260924 17:41:54.047591 16474 consensus_queue.cc:237] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:41:54.048399 16279 catalog_manager.cc:5495] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 250e1986eaef49048f7056fced5c5d9b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "250e1986eaef49048f7056fced5c5d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } health_report { overall_health: UNKNOWN } } }
I20260924 17:41:54.070927 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.068625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.98.1:35869)
0924 17:41:54.068684 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:54.068686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:54.068695 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:41:54.068742 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:54.068743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:54.068763 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:54.068823 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:54.068936 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.069400 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.070181 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.070376 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.070447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.070645 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:54.070703 (+ 58us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 17:41:54.070835 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53196: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 17:41:54.071076 16476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53196: token expired
I20260924 17:41:54.077561 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.071368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.0.98.62:34549)
0924 17:41:54.071440 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:54.071443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:54.071451 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:54.071497 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:54.071498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:54.071508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:54.071558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:41:54.072051 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.072520 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.073203 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.073398 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.073485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.074738 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:41:54.074740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:41:54.074740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:41:54.074748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:41:54.076387 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:54.076774 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:54.076775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:54.076776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:54.076835 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:41:54.077042 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:41:54.077043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:41:54.077043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:41:54.077161 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:41:54.077274 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:54.077374 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 17:41:54.077457 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 17:41:54.080688 16469 tablet.cc:2369] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:41:54.080960 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.078699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.0.98.1:35869)
0924 17:41:54.078743 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:41:54.078745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:41:54.078753 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:41:54.078775 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:41:54.078776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:41:54.078793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:41:54.078862 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:41:54.078972 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.079490 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.080206 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:41:54.080403 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:41:54.080479 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:41:54.080629 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:41:54.080726 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:41:54.080784 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:41:54.080873 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:41:54.080897 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:41:54.082412 16697 raft_consensus.cc:1261] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Refusing update from remote peer 250e1986eaef49048f7056fced5c5d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:54.082412 16562 raft_consensus.cc:1261] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Refusing update from remote peer 250e1986eaef49048f7056fced5c5d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:41:54.082727 16474 consensus_queue.cc:1035] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:54.082921 16756 consensus_queue.cc:1035] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:41:54.085276 16760 mvcc.cc:205] Tried to move back new op lower bound from 7332952940879233024 to 7332952940739633152. Current Snapshot: MvccSnapshot[applied={T|T < 7332952940879233024}]
I20260924 17:41:54.085418 16759 mvcc.cc:205] Tried to move back new op lower bound from 7332952940879233024 to 7332952940739633152. Current Snapshot: MvccSnapshot[applied={T|T < 7332952940879233024 or (T in {7332952940879233024})}]
I20260924 17:41:54.085662 16762 mvcc.cc:205] Tried to move back new op lower bound from 7332952940879233024 to 7332952940739633152. Current Snapshot: MvccSnapshot[applied={T|T < 7332952940879233024 or (T in {7332952940879233024})}]
I20260924 17:41:54.087342 16280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:54.087455 16280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:41:54.088172 16280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Sending DeleteTablet for 3 replicas of tablet 0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:41:54.088524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16319
I20260924 17:41:54.088537 16677 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60468
I20260924 17:41:54.088593 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260924 17:41:54.088680 16542 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8c55007d2f4edc8b3f158b8db040ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:41:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60866
I20260924 17:41:54.089097 16614 tablet_replica.cc:330] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: stopping tablet replica
I20260924 17:41:54.089149 16614 raft_consensus.cc:2229] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:54.089197 16614 raft_consensus.cc:2258] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:54.089605 16769 tablet_replica.cc:330] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: stopping tablet replica
I20260924 17:41:54.089720 16769 raft_consensus.cc:2229] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:41:54.089738 16614 ts_tablet_manager.cc:1905] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:54.089838 16769 raft_consensus.cc:2258] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:41:54.090305 16769 ts_tablet_manager.cc:1905] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:41:54.091605 16614 ts_tablet_manager.cc:1918] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:54.091674 16614 log.cc:1196] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:41:54.091914 16614 ts_tablet_manager.cc:1939] T 0f8c55007d2f4edc8b3f158b8db040ea P cb948705f88640e8b63a6db1f7656f3f: Deleting consensus metadata
I20260924 17:41:54.092262 16264 catalog_manager.cc:4841] TS cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): tablet 0f8c55007d2f4edc8b3f158b8db040ea (table test-table [id=7bff420118c54e67826fd2813d4667b1]) successfully deleted
I20260924 17:41:54.092315 16769 ts_tablet_manager.cc:1918] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:41:54.092371 16769 log.cc:1196] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:41:54.092577 16769 ts_tablet_manager.cc:1939] T 0f8c55007d2f4edc8b3f158b8db040ea P 9d806dd6da1e4a4693e02d805d53640c: Deleting consensus metadata
I20260924 17:41:54.093086 16265 catalog_manager.cc:4841] TS 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): tablet 0f8c55007d2f4edc8b3f158b8db040ea (table test-table [id=7bff420118c54e67826fd2813d4667b1]) successfully deleted
W20260924 17:41:54.095527 16264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35869 (error 108)
I20260924 17:41:54.095839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16473
I20260924 17:41:54.096503 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:41:54.095971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.0.1:53224)
0924 17:41:54.096420 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 17:41:54.096460 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
W20260924 17:41:54.096627 16264 catalog_manager.cc:4584] TS 250e1986eaef49048f7056fced5c5d9b (127.0.98.1:35869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f8c55007d2f4edc8b3f158b8db040ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35869: connect: Connection refused (error 111)
I20260924 17:41:54.101783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16612
I20260924 17:41:54.106657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16247
I20260924 17:42:14.112869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--webserver_interface=127.0.98.62
--webserver_port=43401
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:14.198114 16771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:14.198333 16771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:14.198356 16771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:14.199587 16771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:42:14.199630 16771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:42:14.199642 16771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:42:14.199654 16771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:14.199668 16771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:14.199681 16771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:14.199693 16771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:14.199705 16771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:14.201140 16771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:14.201341 16771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:14.201512 16771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:42:14.206949 16771 server_base.cc:1034] running on GCE node
W20260924 17:42:14.206907 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.206876 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.206887 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:14.207425 16771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:14.207782 16771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:14.208954 16771 hybrid_clock.cc:648] HybridClock initialized: now 1790271734208944 us; error 47 us; skew 500 ppm
Sep 24 17:42:14 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:14.217661 16771 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:14.218206 16771 webserver.cc:460] Webserver started at http://127.0.98.62:43401/ using document root <none> and password file <none>
I20260924 17:42:14.218456 16771 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:14.218537 16771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:14.220006 16771 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:14.220811 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.221036 16771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:14.221149 16771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f7c320746dab4013832325a3a6c1bf50"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "16b8d28f38c0ed166cc0e4ba19da5f5f"
server_key_iv: "a961ac17fdc00b1cf0b2a4eaa3f3e12d"
server_key_version: "encryptionkey@0"
I20260924 17:42:14.221534 16771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:14.234248 16771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:14.236488 16771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:14.236672 16771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:14.240712 16771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34549
I20260924 17:42:14.240767 16838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34549 every 8 connection(s)
I20260924 17:42:14.241163 16771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:14.242626 16839 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:42:14.243291 16839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap starting.
I20260924 17:42:14.248422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16771
I20260924 17:42:14.248715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35869
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=40675
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:42:14.252941 16839 log.cc:824] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:14.255326 16839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:42:14.255649 16839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: Bootstrap complete.
I20260924 17:42:14.257907 16839 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:42:14.258133 16839 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Initialized, Role: FOLLOWER
I20260924 17:42:14.258299 16839 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:42:14.258368 16839 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:14.258430 16839 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:14.258477 16839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 2 FOLLOWER]: Advancing to term 3
I20260924 17:42:14.259585 16839 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:42:14.259675 16839 leader_election.cc:302] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c320746dab4013832325a3a6c1bf50; no voters:
I20260924 17:42:14.259898 16839 leader_election.cc:288] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 17:42:14.259940 16843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 17:42:14.260093 16843 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [term 3 LEADER]: Becoming Leader. State: Replica: f7c320746dab4013832325a3a6c1bf50, State: Running, Role: LEADER
I20260924 17:42:14.260181 16839 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:14.260239 16843 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260924 17:42:14.260730 16845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c320746dab4013832325a3a6c1bf50. Latest consensus state: current_term: 3 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:42:14.260743 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f7c320746dab4013832325a3a6c1bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c320746dab4013832325a3a6c1bf50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260924 17:42:14.260867 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:14.260823 16845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:14.262145 16859 catalog_manager.cc:1259] Loaded cluster ID: 719b21d9978b43a695602a0dc7367824
I20260924 17:42:14.262197 16859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: loading cluster ID for follower catalog manager: success
I20260924 17:42:14.263362 16859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7c320746dab4013832325a3a6c1bf50: acquiring CA information for follower catalog manager: success
F20260924 17:42:14.263751 16859 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f418b2329ad google::LogMessage::Fail() at ??:0
@ 0x7f418b23588a google::LogMessage::SendToLog() at ??:0
@ 0x7f418b2324de google::LogMessage::Flush() at ??:0
@ 0x7f418b233a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f418d99f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f419318b6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f419318b989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f419318be4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f419318e8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f418d0f5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f418df596db start_thread at ??:0
@ 0x7f4189d3061f clone at ??:0
W20260924 17:42:14.346539 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:14.346760 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:14.346786 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:14.348106 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:14.348160 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:14.348188 16841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:14.349740 16841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40675
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:14.349992 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:14.350253 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:14.355367 16877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.355454 16880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:14.355650 16841 server_base.cc:1034] running on GCE node
W20260924 17:42:14.355367 16878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:14.355894 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:14.356200 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:14.357369 16841 hybrid_clock.cc:648] HybridClock initialized: now 1790271734357357 us; error 28 us; skew 500 ppm
Sep 24 17:42:14 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:14.360360 16841 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:42:14.360908 16841 webserver.cc:460] Webserver started at http://127.0.98.1:40675/ using document root <none> and password file <none>
I20260924 17:42:14.361191 16841 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:14.361261 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:14.363092 16841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:14.363973 16887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.364210 16841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:14.364308 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "250e1986eaef49048f7056fced5c5d9b"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "e59d6c51e807ce991bd2f101c9839a3b"
server_key_iv: "6bfdb8aacf7e51c1fc4777e320d83a17"
server_key_version: "encryptionkey@0"
I20260924 17:42:14.364753 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:14.376372 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:14.378995 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:14.379194 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:14.379467 16841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:14.379990 16894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:42:14.381603 16841 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:42:14.381669 16841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:42:14.381705 16841 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:42:14.382651 16841 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:42:14.382697 16841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:42:14.382808 16894 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Bootstrap starting.
I20260924 17:42:14.389873 16841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35869
I20260924 17:42:14.390075 17001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35869 every 8 connection(s)
I20260924 17:42:14.390345 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:14.393555 16894 log.cc:824] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:14.394143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 16841
I20260924 17:42:14.394433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38477
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=34175
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:42:14.396252 16894 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:42:14.396579 16894 tablet_bootstrap.cc:492] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Bootstrap complete.
I20260924 17:42:14.397589 16894 ts_tablet_manager.cc:1397] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260924 17:42:14.399498 16894 raft_consensus.cc:348] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:14.399917 16894 raft_consensus.cc:729] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e1986eaef49048f7056fced5c5d9b, State: Initialized, Role: FOLLOWER
I20260924 17:42:14.400110 16894 consensus_queue.cc:260] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:14.400422 16894 ts_tablet_manager.cc:1428] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 17:42:14.461823 17004 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:42:14.391615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:51945)
0924 17:42:14.391837 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.391902 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:14.392639 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:14.461608 (+ 68969us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
W20260924 17:42:14.462136 17002 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34549: Client connection negotiation failed: client connection to 127.0.98.62:34549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:42:14.462507 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.462380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:52175)
0924 17:42:14.462437 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.462480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:42:14.462889 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.462803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:44153)
0924 17:42:14.462851 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.462873 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:42:14.463059 17002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:42:14.482357 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:14.482563 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:14.482589 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:14.483830 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:14.483876 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:14.483904 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:14.485308 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34175
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:14.485502 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:14.485689 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:14.490579 17015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.490588 17012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.490664 17013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:14.491074 17006 server_base.cc:1034] running on GCE node
I20260924 17:42:14.491288 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:14.491487 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:14.492650 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790271734492628 us; error 35 us; skew 500 ppm
Sep 24 17:42:14 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:14.495129 17006 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:42:14.495589 17006 webserver.cc:460] Webserver started at http://127.0.98.2:34175/ using document root <none> and password file <none>
I20260924 17:42:14.495829 17006 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:14.495898 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:14.497212 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:14.497822 17022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.498001 17006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.498097 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "cb948705f88640e8b63a6db1f7656f3f"
format_stamp: "Formatted at 2026-09-24 17:41:32 on dist-test-slave-d0pf"
server_key: "dda3d58a401475ec2b4ef3ff5e5ebbbb"
server_key_iv: "f56495509db96892bf438d7d0b261269"
server_key_version: "encryptionkey@0"
I20260924 17:42:14.498447 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:14.509006 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:14.511112 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:14.511276 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:14.511529 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:14.511847 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:14.511898 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.511940 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:14.511970 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.518102 17006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38477
I20260924 17:42:14.518142 17135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38477 every 8 connection(s)
I20260924 17:42:14.518537 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:42:14.519709 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.519321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:42613)
0924 17:42:14.519483 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.519560 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
W20260924 17:42:14.520129 17136 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34549: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
I20260924 17:42:14.520439 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.520347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:36409)
0924 17:42:14.520396 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.520422 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:42:14.520686 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.520638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:39455)
0924 17:42:14.520658 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.520675 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260924 17:42:14.520809 17136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:42:14.521258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17006
I20260924 17:42:14.521544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43953
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34549
--webserver_port=39513
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:14.603434 17139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:14.603652 17139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:14.603677 17139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:14.604892 17139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:14.604936 17139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:14.604964 17139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:14.606467 17139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39513
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:14.606734 17139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:14.607120 17139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:14.612401 17145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.612408 17147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:14.612408 17144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:14.612483 17139 server_base.cc:1034] running on GCE node
I20260924 17:42:14.612782 17139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:14.613050 17139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:14.614221 17139 hybrid_clock.cc:648] HybridClock initialized: now 1790271734614206 us; error 35 us; skew 500 ppm
Sep 24 17:42:14 dist-test-slave-d0pf krb5kdc[15703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:14.616883 17139 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:42:14.617396 17139 webserver.cc:460] Webserver started at http://127.0.98.3:39513/ using document root <none> and password file <none>
I20260924 17:42:14.617643 17139 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:14.617714 17139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:14.619192 17139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:14.619822 17154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.620004 17139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:14.620113 17139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "9d806dd6da1e4a4693e02d805d53640c"
format_stamp: "Formatted at 2026-09-24 17:41:33 on dist-test-slave-d0pf"
server_key: "114aced41fb595eac539274eb600d338"
server_key_iv: "3756b7f15ca81b5d7ab8eb22ebf9d7bf"
server_key_version: "encryptionkey@0"
I20260924 17:42:14.620481 17139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:14.631806 17139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:14.633891 17139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:14.634032 17139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:14.634248 17139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:14.634573 17139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:14.634615 17139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.634665 17139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:14.634690 17139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:14.640862 17139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43953
I20260924 17:42:14.640902 17267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43953 every 8 connection(s)
I20260924 17:42:14.641258 17139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:42:14.642307 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.641944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:59345)
0924 17:42:14.642092 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.642166 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
W20260924 17:42:14.642797 17268 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34549: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
I20260924 17:42:14.643195 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.643074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:60201)
0924 17:42:14.643129 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.643171 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:42:14.643496 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:14.643407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:37459)
0924 17:42:14.643453 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:14.643478 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:42:14.643620 17268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:42:14.646735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17139
I20260924 17:42:15.463677 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:15.463440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:47763)
0924 17:42:15.463572 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:15.463613 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 17:42:15.521370 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:15.521158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:46733)
0924 17:42:15.521284 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:15.521322 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 17:42:15.644140 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:15.643923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:35701)
0924 17:42:15.644052 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:15.644089 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260924 17:42:16.196039 17275 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:16.196195 17275 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:16.196583 17275 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
Sep 24 17:42:16 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 17:42:16 dist-test-slave-d0pf krb5kdc[15703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 17:42:16.207978 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.196907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51995 (local address 127.0.98.3:43953)
0924 17:42:16.197250 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:16.197258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:16.197279 (+ 21us) server_negotiation.cc:407] Connection header received
0924 17:42:16.197717 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:16.197719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:16.197763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:16.197925 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:16.199135 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.199960 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201151 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.201657 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201779 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:16.204186 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:16.204191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:16.204191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:16.204205 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:16.205992 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206671 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:16.206674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:16.206674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:16.206760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:16.207018 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:16.207019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:16.207019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:16.207307 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:16.207409 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:16.207725 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 17:42:16.207855 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":71,"threads_started":1}
I20260924 17:42:16.207965 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.196747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43953 (local address 127.0.98.1:51995)
0924 17:42:16.196911 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:16.196922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:16.196986 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:16.198079 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:16.198083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:16.198682 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:16.198883 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:16.198889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.200061 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.200063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:16.201037 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:16.201040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.201745 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:16.201821 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:16.202479 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:16.202489 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:16.204012 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:16.206110 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:16.206119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:16.206348 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:16.206833 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:16.206834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:16.206943 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:16.207389 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:16.207395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:16.207533 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 17:42:16.207656 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 17:42:16.207860 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":87,"thread_start_us":55,"threads_started":1}
I20260924 17:42:16.207990 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.197080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51513 (local address 127.0.98.2:38477)
0924 17:42:16.197239 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:16.197253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:16.197292 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:42:16.197946 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:16.197949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:16.198000 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:16.198125 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:16.199006 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.199921 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201157 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.201588 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201732 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:16.204519 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:16.204523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:16.204523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:16.204566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:16.206338 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206661 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:16.206663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:16.206664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:16.206753 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:16.207016 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:16.207017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:16.207017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:16.207307 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:16.207408 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:16.207704 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 17:42:16.207854 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260924 17:42:16.208088 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.196778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38477 (local address 127.0.98.1:51513)
0924 17:42:16.196931 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:16.196943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:16.197000 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:16.198135 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:16.198136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:16.198642 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:16.198883 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:16.198889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.200036 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.200037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:16.201037 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:16.201045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:16.201704 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:16.201705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:16.201819 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:16.202562 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:16.202567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:16.204371 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:16.206427 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:16.206428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:16.206590 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:16.206853 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:16.206854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:16.206854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:16.206945 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:16.207389 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:16.207396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:16.207533 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 17:42:16.207650 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:42:16.207949 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":192,"thread_start_us":89,"threads_started":1}
W20260924 17:42:16.208978 16888 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
W20260924 17:42:16.209196 16889 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:42:16.209270 16889 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b; no voters: 9d806dd6da1e4a4693e02d805d53640c, cb948705f88640e8b63a6db1f7656f3f
I20260924 17:42:16.209426 17275 raft_consensus.cc:2726] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:42:16.464308 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.464177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:44723)
0924 17:42:16.464233 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:16.464269 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 17:42:16.522228 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.521921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:58137)
0924 17:42:16.522101 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:16.522162 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260924 17:42:16.644899 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:16.644624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:44691)
0924 17:42:16.644770 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:16.644855 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 17:42:17.464967 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:17.464757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:54789)
0924 17:42:17.464877 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:17.464915 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 17:42:17.522904 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:17.522674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:47283)
0924 17:42:17.522815 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:17.522858 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260924 17:42:17.645668 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:17.645399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:33667)
0924 17:42:17.645577 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:17.645621 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260924 17:42:18.341722 17285 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:18.341813 17285 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:18.341974 17285 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
W20260924 17:42:18.342428 16888 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
W20260924 17:42:18.342512 16889 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:42:18.342563 16889 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b; no voters: 9d806dd6da1e4a4693e02d805d53640c, cb948705f88640e8b63a6db1f7656f3f
I20260924 17:42:18.342653 17285 raft_consensus.cc:2726] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:42:18.465746 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:18.465497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:43649)
0924 17:42:18.465640 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:18.465683 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 17:42:18.523608 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:18.523368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:45469)
0924 17:42:18.523516 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:18.523552 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260924 17:42:18.646502 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:18.646242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:33665)
0924 17:42:18.646398 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:18.646444 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 17:42:19.466755 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:19.466451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:43113)
0924 17:42:19.466655 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:19.466705 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
I20260924 17:42:19.524295 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:19.524095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:50117)
0924 17:42:19.524206 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:19.524251 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260924 17:42:19.647296 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:19.647082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:52695)
0924 17:42:19.647201 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:19.647247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260924 17:42:20.467680 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:20.467453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:51283)
0924 17:42:20.467590 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:20.467634 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260924 17:42:20.524955 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:20.524744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:57235)
0924 17:42:20.524874 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:20.524913 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260924 17:42:20.648008 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:20.647816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:48113)
0924 17:42:20.647925 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:20.647958 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260924 17:42:20.799546 17295 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:20.799639 17295 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:20.799803 17295 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
W20260924 17:42:20.800190 16888 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
W20260924 17:42:20.800266 16889 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:42:20.800359 16889 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b; no voters: 9d806dd6da1e4a4693e02d805d53640c, cb948705f88640e8b63a6db1f7656f3f
I20260924 17:42:20.800464 17295 raft_consensus.cc:2726] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:42:21.468425 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:21.468207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:57715)
0924 17:42:21.468334 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:21.468378 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 17:42:21.525664 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:21.525453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:48919)
0924 17:42:21.525577 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:21.525614 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 17:42:21.648823 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:21.648559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:38151)
0924 17:42:21.648725 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:21.648758 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260924 17:42:22.469326 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:22.469048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:53155)
0924 17:42:22.469234 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:22.469278 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260924 17:42:22.526432 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:22.526180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:36051)
0924 17:42:22.526338 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:22.526382 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260924 17:42:22.649545 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:22.649317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:50023)
0924 17:42:22.649444 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:22.649487 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 17:42:23.470021 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:23.469822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:42795)
0924 17:42:23.469936 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:23.469967 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 17:42:23.527261 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:23.526950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:39697)
0924 17:42:23.527170 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:23.527213 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":67,"threads_started":1}
I20260924 17:42:23.650273 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:23.650052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:43533)
0924 17:42:23.650182 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:23.650224 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260924 17:42:24.470702 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:24.470492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:48303)
0924 17:42:24.470613 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:24.470651 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 17:42:24.528013 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:24.527769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:51755)
0924 17:42:24.527909 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:24.527953 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 17:42:24.650987 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:24.650754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:47137)
0924 17:42:24.650878 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:24.650918 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260924 17:42:25.150663 17308 raft_consensus.cc:482] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:25.150774 17308 raft_consensus.cc:504] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d806dd6da1e4a4693e02d805d53640c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43953 } } peers { permanent_uuid: "250e1986eaef49048f7056fced5c5d9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } } peers { permanent_uuid: "cb948705f88640e8b63a6db1f7656f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38477 } }
I20260924 17:42:25.150995 17308 leader_election.cc:288] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953), cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477)
W20260924 17:42:25.151364 16889 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d806dd6da1e4a4693e02d805d53640c (127.0.98.3:43953): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
W20260924 17:42:25.151448 16888 leader_election.cc:341] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb948705f88640e8b63a6db1f7656f3f (127.0.98.2:38477): Not found: Tablet not found: 0f8c55007d2f4edc8b3f158b8db040ea
I20260924 17:42:25.151501 16889 leader_election.cc:302] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250e1986eaef49048f7056fced5c5d9b; no voters: 9d806dd6da1e4a4693e02d805d53640c, cb948705f88640e8b63a6db1f7656f3f
I20260924 17:42:25.151599 17308 raft_consensus.cc:2726] T 0f8c55007d2f4edc8b3f158b8db040ea P 250e1986eaef49048f7056fced5c5d9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:42:25.471382 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:25.471161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:60965)
0924 17:42:25.471301 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:25.471337 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260924 17:42:25.528767 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:25.528519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:33193)
0924 17:42:25.528660 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:25.528714 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260924 17:42:25.651794 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:25.651518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:44597)
0924 17:42:25.651679 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:25.651719 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260924 17:42:26.472163 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:26.471909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:37785)
0924 17:42:26.472071 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:26.472115 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260924 17:42:26.529633 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:26.529373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:42087)
0924 17:42:26.529532 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:26.529582 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260924 17:42:26.652467 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:26.652237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:52695)
0924 17:42:26.652381 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:26.652419 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260924 17:42:27.472971 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:27.472708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:33827)
0924 17:42:27.472835 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:27.472912 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260924 17:42:27.530347 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:27.530134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:41377)
0924 17:42:27.530266 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:27.530304 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 17:42:27.653149 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:27.652936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:50435)
0924 17:42:27.653067 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:27.653101 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260924 17:42:28.473728 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:28.473473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:33155)
0924 17:42:28.473635 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:28.473681 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260924 17:42:28.531131 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:28.530849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:56925)
0924 17:42:28.530980 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:28.531021 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260924 17:42:28.653918 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:28.653672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:54561)
0924 17:42:28.653823 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:28.653869 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260924 17:42:29.474562 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:29.474337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:52093)
0924 17:42:29.474471 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:29.474512 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 17:42:29.531862 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:29.531655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:38405)
0924 17:42:29.531780 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:29.531818 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260924 17:42:29.649118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 16841
I20260924 17:42:29.653757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17006
I20260924 17:42:29.654796 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:29.654579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:40913)
0924 17:42:29.654706 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:29.654737 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260924 17:42:29.658149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17139
2026-09-24T17:42:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58288 ms)
[----------] 30 tests from SecurityITest (233258 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:29 dist-test-slave-d0pf krb5kdc[17326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:29 dist-test-slave-d0pf krb5kdc[17326](info): set up 2 sockets
Sep 24 17:42:29 dist-test-slave-d0pf krb5kdc[17326](info): commencing operation
krb5kdc: starting...
W20260924 17:42:30.702754 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:30 dist-test-slave-d0pf krb5kdc[17326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:30.831007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:30.912771 17339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:30.912977 17339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:30.913002 17339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:30.914199 17339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:30.914244 17339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:30.914260 17339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:30.914273 17339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:30.914285 17339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:30.915956 17339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:30.916216 17339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:30.916460 17339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:30.921847 17344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:30.921820 17345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:30.921918 17339 server_base.cc:1034] running on GCE node
W20260924 17:42:30.921832 17347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:30.922292 17339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:30.922590 17339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:30.923751 17339 hybrid_clock.cc:648] HybridClock initialized: now 1790271750923738 us; error 31 us; skew 500 ppm
Sep 24 17:42:30 dist-test-slave-d0pf krb5kdc[17326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:30.926275 17339 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:30.926712 17339 webserver.cc:460] Webserver started at http://127.0.98.62:39641/ using document root <none> and password file <none>
I20260924 17:42:30.926954 17339 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:30.927022 17339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:30.927197 17339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:30.928259 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "b53725bfef9b4d6099e76115f6d74fe9"
format_stamp: "Formatted at 2026-09-24 17:42:30 on dist-test-slave-d0pf"
server_key: "e321d772053a0f50c430f682201cda39"
server_key_iv: "186c2ae9ebbdd06c27b44b6a83ded239"
server_key_version: "encryptionkey@0"
I20260924 17:42:30.928627 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "b53725bfef9b4d6099e76115f6d74fe9"
format_stamp: "Formatted at 2026-09-24 17:42:30 on dist-test-slave-d0pf"
server_key: "e321d772053a0f50c430f682201cda39"
server_key_iv: "186c2ae9ebbdd06c27b44b6a83ded239"
server_key_version: "encryptionkey@0"
I20260924 17:42:30.929944 17339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:30.930684 17354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:30.930845 17339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:30.930943 17339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "b53725bfef9b4d6099e76115f6d74fe9"
format_stamp: "Formatted at 2026-09-24 17:42:30 on dist-test-slave-d0pf"
server_key: "e321d772053a0f50c430f682201cda39"
server_key_iv: "186c2ae9ebbdd06c27b44b6a83ded239"
server_key_version: "encryptionkey@0"
I20260924 17:42:30.931023 17339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:30.952040 17339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:30.954030 17339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:30.954210 17339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:30.958254 17339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37881
I20260924 17:42:30.958305 17406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37881 every 8 connection(s)
I20260924 17:42:30.958678 17339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:30.959614 17407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:30.961921 17407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: Bootstrap starting.
I20260924 17:42:30.962532 17407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:30.962829 17407 log.cc:824] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:30.963639 17407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: No bootstrap required, opened a new log
I20260924 17:42:30.965277 17407 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260924 17:42:30.965436 17407 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:30.965461 17407 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53725bfef9b4d6099e76115f6d74fe9, State: Initialized, Role: FOLLOWER
I20260924 17:42:30.965643 17407 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260924 17:42:30.965708 17407 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:30.965729 17407 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:30.965811 17407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:30.966239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17339
I20260924 17:42:30.966329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:30.966424 17407 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260924 17:42:30.966557 392 external_mini_cluster.cc:1428] Setting key c90bfd582f10257aee1adca80a36f013
I20260924 17:42:30.966548 17407 leader_election.cc:302] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53725bfef9b4d6099e76115f6d74fe9; no voters:
I20260924 17:42:30.966737 17407 leader_election.cc:288] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:30.966768 17410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:30.966881 17410 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [term 1 LEADER]: Becoming Leader. State: Replica: b53725bfef9b4d6099e76115f6d74fe9, State: Running, Role: LEADER
I20260924 17:42:30.967022 17410 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260924 17:42:30.967015 17407 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:30.967555 17411 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53725bfef9b4d6099e76115f6d74fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } } }
I20260924 17:42:30.967644 17411 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:30.967792 17412 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53725bfef9b4d6099e76115f6d74fe9. Latest consensus state: current_term: 1 leader_uuid: "b53725bfef9b4d6099e76115f6d74fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53725bfef9b4d6099e76115f6d74fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } } }
I20260924 17:42:30.967888 17412 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:30.967835 17416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:30.968467 17416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:30.970108 17416 catalog_manager.cc:1347] Generated new cluster ID: eaa2b7cc8bc1423b8e59f764f3c20b6b
I20260924 17:42:30.970158 17416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:42:30 dist-test-slave-d0pf krb5kdc[17326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:30.977347 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:30.967190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.98.62:37881)
0924 17:42:30.967352 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:30.967362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:30.967398 (+ 36us) server_negotiation.cc:407] Connection header received
0924 17:42:30.967596 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:30.967603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:30.967661 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:30.968658 (+ 997us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:30.968844 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:30.970221 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:30.971010 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:30.971473 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:30.971605 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:30.973904 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:30.973908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:30.973909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:30.973920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:30.975687 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:30.976082 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:30.976085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:30.976085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:30.976188 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:30.976403 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:30.976403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:30.976404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:30.976650 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:30.976823 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:30.976928 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 17:42:30.977178 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260924 17:42:30.981910 17416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:30.982590 17416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:30.985078 17416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: Generated new TSK 0
I20260924 17:42:30.985267 17416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:42:31.012051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:31.094627 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:31.094848 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:31.094875 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:31.096205 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:31.096251 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:31.096279 17433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:31.097819 17433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:31.098069 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:31.098331 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:31.104061 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.104043 17439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.104043 17441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.104665 17433 server_base.cc:1034] running on GCE node
I20260924 17:42:31.104914 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:31.105186 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:31.106367 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790271751106343 us; error 33 us; skew 500 ppm
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:31.108981 17433 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:42:31.109464 17433 webserver.cc:460] Webserver started at http://127.0.98.1:37013/ using document root <none> and password file <none>
I20260924 17:42:31.109653 17433 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:31.109694 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:31.109786 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:31.110821 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "c03a3bed13484118aa339e9256c3edd5"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "225fdc83708f969ed0c6b56465576c80"
server_key_iv: "c9d2fe3a6b30a00a5c2808ed25c849c7"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.111210 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c03a3bed13484118aa339e9256c3edd5"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "225fdc83708f969ed0c6b56465576c80"
server_key_iv: "c9d2fe3a6b30a00a5c2808ed25c849c7"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.112666 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:31.113416 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.113626 17433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:31.113742 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "c03a3bed13484118aa339e9256c3edd5"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "225fdc83708f969ed0c6b56465576c80"
server_key_iv: "c9d2fe3a6b30a00a5c2808ed25c849c7"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.113822 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:31.145754 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:31.148234 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:31.148382 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:31.148663 17433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:31.149003 17433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:31.149060 17433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.149102 17433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:31.149129 17433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.155236 17433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37669
I20260924 17:42:31.155308 17561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37669 every 8 connection(s)
I20260924 17:42:31.155656 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:31.157117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17433
I20260924 17:42:31.157256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:31.157454 392 external_mini_cluster.cc:1428] Setting key 0875f6a95aa5bcb4faec9f4e4f7d46aa
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 17:42:31.166170 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.156484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45435 (local address 127.0.98.62:37881)
0924 17:42:31.156554 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.156556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.156775 (+ 219us) server_negotiation.cc:407] Connection header received
0924 17:42:31.157935 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.157936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.157948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.158003 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.158703 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.159397 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.160391 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.160727 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.160803 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.162973 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.162975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.162975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:31.162984 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.164731 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.165128 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.165129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.165130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.165189 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.165517 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.165518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.165518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.165673 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.165742 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.166027 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.166102 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:42:31.166339 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.156454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.1:45435)
0924 17:42:31.156653 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.156691 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.157827 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.158074 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.158079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.158396 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:31.158607 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.158612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.159480 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.159481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.160305 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.160306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.160813 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.160814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.160914 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.161512 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:31.161521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:31.162873 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:31.164825 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.164831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.164833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.165049 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.165284 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.165285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.165285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.165434 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.165764 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:31.165770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:31.165898 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.165989 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.166113 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20260924 17:42:31.166918 17562 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260924 17:42:31.167019 17562 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:31.167258 17562 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260924 17:42:31.167903 17371 ts_manager.cc:194] Registered new tserver with Master: c03a3bed13484118aa339e9256c3edd5 (127.0.98.1:37669)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 17:42:31.168663 17371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:42:31.182050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:31.264845 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:31.265058 17567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:31.265084 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:31.266247 17567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:31.266285 17567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:31.266312 17567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:31.267927 17567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:31.268181 17567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:31.268373 17567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:31.273296 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.273335 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.273388 17567 server_base.cc:1034] running on GCE node
W20260924 17:42:31.273314 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.273804 17567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:31.274051 17567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:31.275189 17567 hybrid_clock.cc:648] HybridClock initialized: now 1790271751275166 us; error 39 us; skew 500 ppm
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:31.277632 17567 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:42:31.278085 17567 webserver.cc:460] Webserver started at http://127.0.98.2:34063/ using document root <none> and password file <none>
I20260924 17:42:31.278316 17567 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:31.278383 17567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:31.278542 17567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:31.279632 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "71b729a46aea4652b4300b16ef4318c9"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "130ca40ba78822d03a098cdd2b84ff21"
server_key_iv: "253d339f52da8ad59b691d37f3840781"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.279987 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71b729a46aea4652b4300b16ef4318c9"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "130ca40ba78822d03a098cdd2b84ff21"
server_key_iv: "253d339f52da8ad59b691d37f3840781"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.281265 17567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:31.281966 17582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.282135 17567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:31.282232 17567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "71b729a46aea4652b4300b16ef4318c9"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "130ca40ba78822d03a098cdd2b84ff21"
server_key_iv: "253d339f52da8ad59b691d37f3840781"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.282311 17567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:31.298421 17567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:31.300571 17567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:31.300731 17567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:31.300973 17567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:31.301270 17567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:31.301322 17567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.301367 17567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:31.301401 17567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.307444 17567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41897
I20260924 17:42:31.307530 17695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41897 every 8 connection(s)
I20260924 17:42:31.307866 17567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:31.316723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17567
I20260924 17:42:31.316841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:31.317036 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.308620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56551 (local address 127.0.98.62:37881)
0924 17:42:31.308708 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.308710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.308893 (+ 183us) server_negotiation.cc:407] Connection header received
0924 17:42:31.309869 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.309870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.309883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.309947 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.310566 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.311138 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.312163 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.312371 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.312446 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.314434 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.314435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.314436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:31.314443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.315652 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.316071 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.316072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.316073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.316129 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.316370 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.316371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.316371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.316481 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.316558 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.316846 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.316938 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:42:31.317049 392 external_mini_cluster.cc:1428] Setting key 39268e218da208fa1023a6f701aed50b
I20260924 17:42:31.317256 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.308594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.2:56551)
0924 17:42:31.308760 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.308810 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.309748 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.310009 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.310013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.310274 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:31.310462 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.310467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.311228 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.311229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.312071 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.312074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.312464 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.312465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.312581 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.313051 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:31.313059 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:31.314341 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:31.315736 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.315742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.315744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.315988 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.316195 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.316196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.316196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.316303 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.316549 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:31.316555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:31.316713 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.316868 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.317023 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260924 17:42:31.317826 17696 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260924 17:42:31.317958 17696 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:31.318135 17696 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260924 17:42:31.318521 17371 ts_manager.cc:194] Registered new tserver with Master: 71b729a46aea4652b4300b16ef4318c9 (127.0.98.2:41897)
I20260924 17:42:31.318903 17371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56551
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:42:31.339962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:31.419661 17701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:31.419939 17701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:31.419981 17701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:31.421159 17701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:31.421226 17701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:31.421274 17701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:31.422796 17701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:31.423099 17701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:31.423342 17701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:31.428403 17709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.428457 17707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.428464 17706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.428666 17701 server_base.cc:1034] running on GCE node
I20260924 17:42:31.428880 17701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:31.429116 17701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:31.430254 17701 hybrid_clock.cc:648] HybridClock initialized: now 1790271751430233 us; error 40 us; skew 500 ppm
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:31.432725 17701 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:42:31.433231 17701 webserver.cc:460] Webserver started at http://127.0.98.3:37057/ using document root <none> and password file <none>
I20260924 17:42:31.433499 17701 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:31.433568 17701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:31.433701 17701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:31.434854 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "30326132d820445084c07a111b8d30aa"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "3b5d26525a83c955c80df6fe053e998e"
server_key_iv: "4f6412ae882673493741a98d576c65e9"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.435231 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30326132d820445084c07a111b8d30aa"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "3b5d26525a83c955c80df6fe053e998e"
server_key_iv: "4f6412ae882673493741a98d576c65e9"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.436650 17701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:31.437402 17716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.437567 17701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:42:31.437671 17701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "30326132d820445084c07a111b8d30aa"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "3b5d26525a83c955c80df6fe053e998e"
server_key_iv: "4f6412ae882673493741a98d576c65e9"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.437745 17701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:31.458539 17701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:31.460673 17701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:31.460846 17701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:31.461081 17701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:31.461409 17701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:31.461467 17701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.461508 17701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:31.461535 17701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.467702 17701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44437
I20260924 17:42:31.467744 17829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44437 every 8 connection(s)
I20260924 17:42:31.468137 17701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:42:31 dist-test-slave-d0pf krb5kdc[17326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:31.474917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17701
I20260924 17:42:31.475075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:31.475311 392 external_mini_cluster.cc:1428] Setting key 11770c7870a9e37fe227dcd42f14b3a4
I20260924 17:42:31.477705 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.468892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34279 (local address 127.0.98.62:37881)
0924 17:42:31.468959 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.468961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.469206 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:42:31.470111 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.470113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.470123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.470172 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.470773 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.471356 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.472348 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.472528 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.472577 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.474719 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.474721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.474721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:31.474730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.476411 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.476778 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.476779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.476780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.476831 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.476997 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.476997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.476998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.477093 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.477159 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.477548 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.477628 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:42:31.477842 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.468868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.3:34279)
0924 17:42:31.469052 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.469102 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.470020 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.470237 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.470241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.470510 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:31.470703 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.470707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.471431 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.471432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.472247 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.472249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.472597 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.472598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.472702 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.473242 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:31.473251 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:31.474604 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:31.476478 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.476484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.476486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.476713 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.476874 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:31.476875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:31.476875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:31.476962 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:31.477164 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:31.477168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:31.477379 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.477512 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.477643 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260924 17:42:31.478427 17830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260924 17:42:31.478516 17830 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:31.478708 17830 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260924 17:42:31.479140 17371 ts_manager.cc:194] Registered new tserver with Master: 30326132d820445084c07a111b8d30aa (127.0.98.3:44437)
I20260924 17:42:31.479486 17371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34279
I20260924 17:42:31.487427 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:31.495188 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.488852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.0.98.62:37881)
0924 17:42:31.488919 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.488921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.489012 (+ 91us) server_negotiation.cc:407] Connection header received
0924 17:42:31.489056 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.489056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.489065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.489118 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.489713 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.490197 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.490939 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.491186 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.491250 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.492452 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.492453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.492453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:31.492459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.494094 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.494454 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.494455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.494455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.494508 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:31.494641 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:31.494642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:31.494642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:31.494782 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.494857 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.494991 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.495121 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:42:31.496925 17371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:31.498508 17371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:31.505095 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.500997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44437 (local address 127.0.0.1:51532)
0924 17:42:31.501592 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.501598 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.501657 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.501924 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.501925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.501927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:31.501976 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.501977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.502833 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.502834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.503902 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.503905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.504773 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.504774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.504862 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.504875 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.504948 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.505000 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":55,"thread_start_us":82,"threads_started":1}
I20260924 17:42:31.505125 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.501230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.98.3:44437)
0924 17:42:31.501400 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.501405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.501708 (+ 303us) server_negotiation.cc:407] Connection header received
0924 17:42:31.501750 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.501753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.501768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.501837 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:31.502032 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.502719 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.504012 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.504570 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.504640 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.504736 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.504993 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.505037 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260924 17:42:31.505658 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.500967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41897 (local address 127.0.0.1:57986)
0924 17:42:31.501122 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.501182 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.501279 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.501665 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.501668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.501670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:31.501761 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.501763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.503190 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.503191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.504557 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.504559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.505384 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.505385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.505449 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.505455 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.505507 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.505562 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260924 17:42:31.505780 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.501158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.0.98.2:41897)
0924 17:42:31.501455 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.501460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.501471 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:42:31.501511 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.501513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.501552 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.501892 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:31.501979 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.503086 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.504637 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.505278 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.505382 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.505442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.505519 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.505678 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260924 17:42:31.506036 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.501298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.98.1:37669)
0924 17:42:31.501468 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.501471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.501554 (+ 83us) server_negotiation.cc:407] Connection header received
0924 17:42:31.501598 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.501600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.501616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.501690 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:31.501921 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.503077 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.504466 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.505224 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.505298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.505355 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.505902 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.505946 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 17:42:31.506255 17764 tablet_service.cc:1467] Processing CreateTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 (DEFAULT_TABLE table=test-table [id=e80e530b9d244639b637cd896a65fdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:31.506659 17764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f53b065fdb4d709a3f080e9c9c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:31.506850 17630 tablet_service.cc:1467] Processing CreateTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 (DEFAULT_TABLE table=test-table [id=e80e530b9d244639b637cd896a65fdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:31.507164 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f53b065fdb4d709a3f080e9c9c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:31.505733 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.500967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37669 (local address 127.0.0.1:38152)
0924 17:42:31.501470 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.501479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.501541 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.501783 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.501785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.501787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:31.501844 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.501846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.503311 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.503312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.504379 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.504382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.505386 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.505388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.505453 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.505459 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.505508 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.505555 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":4,"thread_start_us":44,"threads_started":1}
I20260924 17:42:31.509467 17847 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Bootstrap starting.
I20260924 17:42:31.509958 17848 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Bootstrap starting.
I20260924 17:42:31.510125 17847 tablet_bootstrap.cc:654] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:31.510318 17496 tablet_service.cc:1467] Processing CreateTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 (DEFAULT_TABLE table=test-table [id=e80e530b9d244639b637cd896a65fdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:31.510584 17848 tablet_bootstrap.cc:654] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:31.510670 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f53b065fdb4d709a3f080e9c9c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:31.510737 17847 log.cc:824] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:31.511286 17848 log.cc:824] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:31.511590 17847 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: No bootstrap required, opened a new log
I20260924 17:42:31.511731 17847 ts_tablet_manager.cc:1397] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:31.512050 17848 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: No bootstrap required, opened a new log
I20260924 17:42:31.512138 17848 ts_tablet_manager.cc:1397] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:42:31.513656 17848 raft_consensus.cc:348] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.513913 17851 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: Bootstrap starting.
I20260924 17:42:31.513933 17848 raft_consensus.cc:374] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:31.513818 17847 raft_consensus.cc:348] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.514002 17847 raft_consensus.cc:374] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:31.513975 17848 raft_consensus.cc:729] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b729a46aea4652b4300b16ef4318c9, State: Initialized, Role: FOLLOWER
I20260924 17:42:31.514092 17847 raft_consensus.cc:729] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30326132d820445084c07a111b8d30aa, State: Initialized, Role: FOLLOWER
I20260924 17:42:31.514148 17848 consensus_queue.cc:260] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.514221 17847 consensus_queue.cc:260] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.514490 17847 ts_tablet_manager.cc:1428] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:31.514484 17848 ts_tablet_manager.cc:1428] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:42:31.514566 17851 tablet_bootstrap.cc:654] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:31.514621 17696 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
I20260924 17:42:31.514628 17830 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
I20260924 17:42:31.515182 17851 log.cc:824] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:31.516048 17851 tablet_bootstrap.cc:492] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: No bootstrap required, opened a new log
I20260924 17:42:31.516120 17851 ts_tablet_manager.cc:1397] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:31.517498 17851 raft_consensus.cc:348] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.517624 17851 raft_consensus.cc:374] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:31.517647 17851 raft_consensus.cc:729] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03a3bed13484118aa339e9256c3edd5, State: Initialized, Role: FOLLOWER
I20260924 17:42:31.517791 17851 consensus_queue.cc:260] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.518079 17851 ts_tablet_manager.cc:1428] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:31.518098 17562 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
W20260924 17:42:31.558478 17697 tablet.cc:2369] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:42:31.590354 17853 raft_consensus.cc:482] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:31.590487 17853 raft_consensus.cc:504] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.590974 17853 leader_election.cc:288] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03a3bed13484118aa339e9256c3edd5 (127.0.98.1:37669), 71b729a46aea4652b4300b16ef4318c9 (127.0.98.2:41897)
I20260924 17:42:31.594033 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.591198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41897 (local address 127.0.98.3:59399)
0924 17:42:31.591260 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.591273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.591323 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.591487 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.591488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.591490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:31.591565 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.591567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.592217 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.592218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.593139 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.593142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.593781 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.593864 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.593868 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.593923 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.593969 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":35}
I20260924 17:42:31.594077 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.591307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59399 (local address 127.0.98.2:41897)
0924 17:42:31.591366 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.591369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.591376 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:31.591409 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.591410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.591420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.591483 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:31.591638 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.592108 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593238 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.593692 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593752 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.593760 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.593943 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.593985 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:42:31.594201 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.591209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37669 (local address 127.0.98.3:53255)
0924 17:42:31.591428 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.591436 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.591514 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.591669 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.591670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.591671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:31.591746 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.591748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.592443 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.592444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.593335 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.593337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.593940 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.594008 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.594012 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.594070 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.594110 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":22,"thread_start_us":116,"threads_started":1}
I20260924 17:42:31.594220 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.591277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53255 (local address 127.0.98.1:37669)
0924 17:42:31.591336 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.591339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.591505 (+ 166us) server_negotiation.cc:407] Connection header received
0924 17:42:31.591582 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.591583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.591597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.591666 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:31.591819 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.592366 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593430 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.593829 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.593914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.593924 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.594093 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.594140 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:42:31.594635 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f53b065fdb4d709a3f080e9c9c5dd0" candidate_uuid: "30326132d820445084c07a111b8d30aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b729a46aea4652b4300b16ef4318c9" is_pre_election: true
I20260924 17:42:31.594645 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f53b065fdb4d709a3f080e9c9c5dd0" candidate_uuid: "30326132d820445084c07a111b8d30aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a3bed13484118aa339e9256c3edd5" is_pre_election: true
I20260924 17:42:31.594794 17516 raft_consensus.cc:2454] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30326132d820445084c07a111b8d30aa in term 0.
I20260924 17:42:31.594796 17650 raft_consensus.cc:2454] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30326132d820445084c07a111b8d30aa in term 0.
I20260924 17:42:31.595068 17718 leader_election.cc:302] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30326132d820445084c07a111b8d30aa, 71b729a46aea4652b4300b16ef4318c9; no voters:
I20260924 17:42:31.595223 17853 raft_consensus.cc:2781] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:31.595280 17853 raft_consensus.cc:482] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:31.595309 17853 raft_consensus.cc:3037] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:31.595992 17853 raft_consensus.cc:504] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.596136 17853 leader_election.cc:288] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [CANDIDATE]: Term 1 election: Requested vote from peers c03a3bed13484118aa339e9256c3edd5 (127.0.98.1:37669), 71b729a46aea4652b4300b16ef4318c9 (127.0.98.2:41897)
I20260924 17:42:31.596307 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f53b065fdb4d709a3f080e9c9c5dd0" candidate_uuid: "30326132d820445084c07a111b8d30aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b729a46aea4652b4300b16ef4318c9"
I20260924 17:42:31.596371 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f53b065fdb4d709a3f080e9c9c5dd0" candidate_uuid: "30326132d820445084c07a111b8d30aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a3bed13484118aa339e9256c3edd5"
I20260924 17:42:31.596423 17650 raft_consensus.cc:3037] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:31.596446 17516 raft_consensus.cc:3037] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:31.597376 17650 raft_consensus.cc:2454] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30326132d820445084c07a111b8d30aa in term 1.
I20260924 17:42:31.597376 17516 raft_consensus.cc:2454] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30326132d820445084c07a111b8d30aa in term 1.
I20260924 17:42:31.597582 17718 leader_election.cc:302] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30326132d820445084c07a111b8d30aa, 71b729a46aea4652b4300b16ef4318c9; no voters:
I20260924 17:42:31.597678 17853 raft_consensus.cc:2781] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:31.597867 17853 raft_consensus.cc:686] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 1 LEADER]: Becoming Leader. State: Replica: 30326132d820445084c07a111b8d30aa, State: Running, Role: LEADER
I20260924 17:42:31.597980 17853 consensus_queue.cc:237] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } }
I20260924 17:42:31.598953 17370 catalog_manager.cc:5495] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 30326132d820445084c07a111b8d30aa (127.0.98.3). New cstate: current_term: 1 leader_uuid: "30326132d820445084c07a111b8d30aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30326132d820445084c07a111b8d30aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44437 } health_report { overall_health: HEALTHY } } }
I20260924 17:42:31.620198 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.617804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.0.98.3:44437)
0924 17:42:31.617865 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.617867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.617876 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:31.617922 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.617922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.617940 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.618001 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:31.618115 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.618600 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.619395 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.619579 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.619632 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.619763 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:31.619862 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:31.619932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.620073 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.620114 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:42:31.621958 17516 raft_consensus.cc:1261] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 1 FOLLOWER]: Refusing update from remote peer 30326132d820445084c07a111b8d30aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:31.621958 17650 raft_consensus.cc:1261] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 1 FOLLOWER]: Refusing update from remote peer 30326132d820445084c07a111b8d30aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:31.622279 17853 consensus_queue.cc:1035] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b729a46aea4652b4300b16ef4318c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:31.622411 17857 consensus_queue.cc:1035] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03a3bed13484118aa339e9256c3edd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:31.624922 17860 mvcc.cc:205] Tried to move back new op lower bound from 7332953094641229824 to 7332953094546411520. Current Snapshot: MvccSnapshot[applied={T|T < 7332953094641229824}]
I20260924 17:42:31.625710 17861 mvcc.cc:205] Tried to move back new op lower bound from 7332953094641229824 to 7332953094546411520. Current Snapshot: MvccSnapshot[applied={T|T < 7332953094641229824 or (T in {7332953094641229824})}]
I20260924 17:42:31.626955 17370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:31.627085 17370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:31.627797 17370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b53725bfef9b4d6099e76115f6d74fe9: Sending DeleteTablet for 3 replicas of tablet 79f53b065fdb4d709a3f080e9c9c5dd0
I20260924 17:42:31.628170 17764 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51532
I20260924 17:42:31.628194 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57986
I20260924 17:42:31.628166 17496 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f53b065fdb4d709a3f080e9c9c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38152
I20260924 17:42:31.628480 17872 tablet_replica.cc:330] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5: stopping tablet replica
I20260924 17:42:31.628661 17872 raft_consensus.cc:2229] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:31.628755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17433
I20260924 17:42:31.628780 17872 raft_consensus.cc:2258] T 79f53b065fdb4d709a3f080e9c9c5dd0 P c03a3bed13484118aa339e9256c3edd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:31.629408 17870 tablet_replica.cc:330] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: stopping tablet replica
I20260924 17:42:31.629503 17870 raft_consensus.cc:2229] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:42:31.629621 17870 raft_consensus.cc:2258] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:31.629796 17871 tablet_replica.cc:330] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9: stopping tablet replica
I20260924 17:42:31.629916 17871 raft_consensus.cc:2229] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:31.630039 17871 raft_consensus.cc:2258] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 71b729a46aea4652b4300b16ef4318c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:31.630137 17870 ts_tablet_manager.cc:1905] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:31.632128 17870 ts_tablet_manager.cc:1918] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:31.632202 17870 log.cc:1196] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/79f53b065fdb4d709a3f080e9c9c5dd0
I20260924 17:42:31.632459 17870 ts_tablet_manager.cc:1939] T 79f53b065fdb4d709a3f080e9c9c5dd0 P 30326132d820445084c07a111b8d30aa: Deleting consensus metadata
I20260924 17:42:31.632833 17355 catalog_manager.cc:4841] TS 30326132d820445084c07a111b8d30aa (127.0.98.3:44437): tablet 79f53b065fdb4d709a3f080e9c9c5dd0 (table test-table [id=e80e530b9d244639b637cd896a65fdbe]) successfully deleted
W20260924 17:42:31.634758 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37669 (error 108)
I20260924 17:42:31.635306 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.635174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37669 (local address 127.0.0.1:38154)
0924 17:42:31.635229 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.635274 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:42:31.635324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17567
W20260924 17:42:31.637943 17357 catalog_manager.cc:4584] TS c03a3bed13484118aa339e9256c3edd5 (127.0.98.1:37669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79f53b065fdb4d709a3f080e9c9c5dd0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37669: connect: Connection refused (error 111)
I20260924 17:42:31.641247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17701
I20260924 17:42:31.641325 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.641217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41897 (local address 127.0.0.1:57998)
0924 17:42:31.641258 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.641289 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 17:42:31.645359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17339
2026-09-24T17:42:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T17:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:31Z Disabled control of system clock
I20260924 17:42:31.672123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45685
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:31.751670 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:31.751886 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:31.751906 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:31.752987 17877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:31.753021 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:31.753034 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:31.753047 17877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:31.753059 17877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:31.754313 17877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:31.754498 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:31.754726 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:42:31.759500 17877 server_base.cc:1034] running on GCE node
W20260924 17:42:31.759433 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.759449 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.759462 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.760015 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:31.760298 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:31.761478 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790271751761467 us; error 38 us; skew 500 ppm
I20260924 17:42:31.762770 17877 webserver.cc:460] Webserver started at http://127.0.98.62:43803/ using document root <none> and password file <none>
I20260924 17:42:31.763017 17877 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:31.763127 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:31.763273 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:31.764355 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "92df56d574524ef49da8922aff532366"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "5151b8cd17db6d6fd015b84c78ab8d3c"
server_key_iv: "0297aaf564ac190104b926b69648b4e4"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.764707 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "92df56d574524ef49da8922aff532366"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "5151b8cd17db6d6fd015b84c78ab8d3c"
server_key_iv: "0297aaf564ac190104b926b69648b4e4"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.766067 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:31.766773 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.766953 17877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:31.767086 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "92df56d574524ef49da8922aff532366"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "5151b8cd17db6d6fd015b84c78ab8d3c"
server_key_iv: "0297aaf564ac190104b926b69648b4e4"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.767168 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:31.777549 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:31.777860 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:31.778026 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:31.781927 17877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45685
I20260924 17:42:31.781973 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45685 every 8 connection(s)
I20260924 17:42:31.782325 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:31.783377 17944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:31.785583 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: Bootstrap starting.
I20260924 17:42:31.785872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17877
I20260924 17:42:31.785984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:31.786163 392 external_mini_cluster.cc:1428] Setting key 7b7b92e73df14745fa3f92665281a716
I20260924 17:42:31.786247 17944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:31.786557 17944 log.cc:824] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:31.787374 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: No bootstrap required, opened a new log
I20260924 17:42:31.789708 17944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } }
I20260924 17:42:31.789845 17944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:31.789867 17944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92df56d574524ef49da8922aff532366, State: Initialized, Role: FOLLOWER
I20260924 17:42:31.790000 17944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } }
I20260924 17:42:31.790074 17944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:31.790115 17944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:31.790161 17944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:31.790709 17944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } }
I20260924 17:42:31.790836 17944 leader_election.cc:302] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92df56d574524ef49da8922aff532366; no voters:
I20260924 17:42:31.791061 17944 leader_election.cc:288] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:31.791095 17949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:31.791193 17949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [term 1 LEADER]: Becoming Leader. State: Replica: 92df56d574524ef49da8922aff532366, State: Running, Role: LEADER
I20260924 17:42:31.791330 17944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:31.791337 17949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } }
I20260924 17:42:31.791751 17951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92df56d574524ef49da8922aff532366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } } }
I20260924 17:42:31.791849 17951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:31.791746 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92df56d574524ef49da8922aff532366. Latest consensus state: current_term: 1 leader_uuid: "92df56d574524ef49da8922aff532366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df56d574524ef49da8922aff532366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45685 } } }
I20260924 17:42:31.792207 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:31.792557 17955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:31.792613 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.786679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.98.62:45685)
0924 17:42:31.786892 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.786899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.786923 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:42:31.787532 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.787540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.787612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.787761 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.787960 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.789381 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.790337 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.790742 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.790905 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.791583 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.791586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.791587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:31.791792 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.791832 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:31.791846 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.792166 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.792199 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.792278 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 17:42:31.792987 17955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:31.794431 17955 catalog_manager.cc:1347] Generated new cluster ID: 4a9c6f459aad470f85be6cbcc7134e2b
I20260924 17:42:31.794482 17955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:31.801795 17955 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:31.802294 17955 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:31.809068 17955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: Generated new TSK 0
I20260924 17:42:31.809242 17955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:31.811990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--builtin_ntp_servers=127.0.98.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:31.892619 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:31.892920 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:31.892961 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:31.894119 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:31.894186 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:31.894227 17968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:31.895596 17968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:31.895943 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:31.896243 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:31.901011 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.901070 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:31.901011 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:31.901234 17968 server_base.cc:1034] running on GCE node
I20260924 17:42:31.901495 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:31.901747 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:31.902917 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790271751902900 us; error 31 us; skew 500 ppm
I20260924 17:42:31.904304 17968 webserver.cc:460] Webserver started at http://127.0.98.1:41409/ using document root <none> and password file <none>
I20260924 17:42:31.904570 17968 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:31.904628 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:31.904754 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:31.906121 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddbdb751c9754e7fb836e40f2a55a28c"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "830c8a17ce50bd82ad525e6ac881c22f"
server_key_iv: "7f5d8c44ae43b3d5ec6d78cb9b22b9c2"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.906507 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddbdb751c9754e7fb836e40f2a55a28c"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "830c8a17ce50bd82ad525e6ac881c22f"
server_key_iv: "7f5d8c44ae43b3d5ec6d78cb9b22b9c2"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.908181 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:42:31.909042 17982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.909237 17968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:31.909337 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "ddbdb751c9754e7fb836e40f2a55a28c"
format_stamp: "Formatted at 2026-09-24 17:42:31 on dist-test-slave-d0pf"
server_key: "830c8a17ce50bd82ad525e6ac881c22f"
server_key_iv: "7f5d8c44ae43b3d5ec6d78cb9b22b9c2"
server_key_version: "encryptionkey@0"
I20260924 17:42:31.909412 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:31.929260 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:31.929641 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:31.929792 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:31.930058 17968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:31.930406 17968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:31.930464 17968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.930505 17968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:31.930534 17968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:31.936806 17968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41653
I20260924 17:42:31.936843 18095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41653 every 8 connection(s)
I20260924 17:42:31.937189 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:31.938076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 17968
I20260924 17:42:31.938186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:31.938381 392 external_mini_cluster.cc:1428] Setting key a926a03de47a97a887787440e2abe805
I20260924 17:42:31.939414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--builtin_ntp_servers=127.0.98.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:42:31.943081 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.938589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35057 (local address 127.0.98.62:45685)
0924 17:42:31.938643 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:31.938645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:31.938658 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:42:31.939017 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:31.939018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:31.939030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:31.939116 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:31.939845 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.940356 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.941444 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.941775 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.941851 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.942698 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:31.942699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:31.942699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:31.942705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:31.942724 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:31.942729 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:31.942790 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:31.942900 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:42:31.942929 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:42:31.943212 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:31.937997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45685 (local address 127.0.98.1:35057)
0924 17:42:31.938196 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:31.938238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:31.938901 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:31.939210 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:31.939214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:31.939225 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:31.939741 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.939746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.940494 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.940495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.941353 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:31.941355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:31.941865 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:31.941866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:31.941973 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:31.942552 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:31.942553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:31.942580 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:31.942603 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:31.942811 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:31.942816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:31.942818 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:42:31.942875 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:42:31.942956 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260924 17:42:31.943754 18096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45685
I20260924 17:42:31.943866 18096 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:31.944090 18096 heartbeater.cc:511] Master 127.0.98.62:45685 requested a full tablet report, sending...
I20260924 17:42:31.944592 17908 ts_manager.cc:194] Registered new tserver with Master: ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653)
I20260924 17:42:31.945216 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35057
W20260924 17:42:32.020991 18099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:32.021216 18099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:32.021239 18099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:32.022436 18099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:32.022478 18099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:32.022504 18099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:32.023883 18099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:32.024143 18099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:32.024451 18099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:32.029474 18104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:32.029481 18105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:32.029489 18107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:32.029553 18099 server_base.cc:1034] running on GCE node
I20260924 17:42:32.029913 18099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:32.030206 18099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:32.031390 18099 hybrid_clock.cc:648] HybridClock initialized: now 1790271752031382 us; error 34 us; skew 500 ppm
I20260924 17:42:32.032749 18099 webserver.cc:460] Webserver started at http://127.0.98.2:44601/ using document root <none> and password file <none>
I20260924 17:42:32.033028 18099 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:32.033106 18099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:32.033260 18099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:32.034415 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "a4d324b506d87280145b17aaed2b237e"
server_key_iv: "79f020763fa977c27283efb904702a52"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.034777 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "a4d324b506d87280145b17aaed2b237e"
server_key_iv: "79f020763fa977c27283efb904702a52"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.036240 18099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:42:32.037047 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.037217 18099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:42:32.037322 18099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "a4d324b506d87280145b17aaed2b237e"
server_key_iv: "79f020763fa977c27283efb904702a52"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.037395 18099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:32.056408 18099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:32.056757 18099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:32.056912 18099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:32.057150 18099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:32.057475 18099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:32.057534 18099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.057577 18099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:32.057602 18099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.063414 18099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35193
I20260924 17:42:32.063455 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35193 every 8 connection(s)
I20260924 17:42:32.063838 18099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:42:32.064523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18099
I20260924 17:42:32.064648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:32.064869 392 external_mini_cluster.cc:1428] Setting key 8ef90e9f2cf258aa3e713d80c7010954
I20260924 17:42:32.065907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--builtin_ntp_servers=127.0.98.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:42:32.069911 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.064693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51275 (local address 127.0.98.62:45685)
0924 17:42:32.064746 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.064749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.065017 (+ 268us) server_negotiation.cc:407] Connection header received
0924 17:42:32.065691 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.065692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.065704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.065749 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:32.066495 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.067151 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.068133 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.068470 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.068549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.069521 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:32.069523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:32.069523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:32.069528 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:32.069543 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:32.069555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:32.069606 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.069786 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.069816 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:42:32.070176 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.064669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45685 (local address 127.0.98.2:51275)
0924 17:42:32.064849 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.064891 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.065575 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.065818 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.065822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.065832 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:32.066402 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.066407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.067247 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.067248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.068047 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.068049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.068584 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.068585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.068712 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.069342 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:32.069343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:32.069387 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:32.069419 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:32.069630 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:32.069637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:32.069640 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.069731 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.069886 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260924 17:42:32.070752 18227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45685
I20260924 17:42:32.070887 18227 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:32.071120 18227 heartbeater.cc:511] Master 127.0.98.62:45685 requested a full tablet report, sending...
I20260924 17:42:32.071564 17908 ts_manager.cc:194] Registered new tserver with Master: 8e2b6296b4ba4e9dab0f46435a35d7ce (127.0.98.2:35193)
I20260924 17:42:32.071941 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51275
W20260924 17:42:32.149235 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:32.149458 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:32.149490 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:32.150736 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:32.150775 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:32.150802 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:32.152220 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:32.152486 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:32.152792 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:32.157846 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:32.157841 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:32.157922 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:32.158286 18230 server_base.cc:1034] running on GCE node
I20260924 17:42:32.158597 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:32.158882 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:32.160079 18230 hybrid_clock.cc:648] HybridClock initialized: now 1790271752160082 us; error 47 us; skew 500 ppm
I20260924 17:42:32.161577 18230 webserver.cc:460] Webserver started at http://127.0.98.3:44175/ using document root <none> and password file <none>
I20260924 17:42:32.161870 18230 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:32.161953 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:32.162130 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:32.163448 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "2066953f32bf48fe9f287343233459b2"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "7b1d3bd1dc3186216d32d925a1d9b57f"
server_key_iv: "551b7f7badc2c67778887a9640ce48e0"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.163830 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2066953f32bf48fe9f287343233459b2"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "7b1d3bd1dc3186216d32d925a1d9b57f"
server_key_iv: "551b7f7badc2c67778887a9640ce48e0"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.165401 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:32.166211 18244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.166410 18230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:32.166524 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "2066953f32bf48fe9f287343233459b2"
format_stamp: "Formatted at 2026-09-24 17:42:32 on dist-test-slave-d0pf"
server_key: "7b1d3bd1dc3186216d32d925a1d9b57f"
server_key_iv: "551b7f7badc2c67778887a9640ce48e0"
server_key_version: "encryptionkey@0"
I20260924 17:42:32.166607 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:32.181488 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:32.181851 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:32.182031 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:32.182298 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:32.182663 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:32.182725 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.182772 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:32.182804 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:32.189124 18230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33975
I20260924 17:42:32.189165 18357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33975 every 8 connection(s)
I20260924 17:42:32.189545 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:42:32.191125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18230
I20260924 17:42:32.191239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:32.191540 392 external_mini_cluster.cc:1428] Setting key 513711fbf61bac0b4718f30f8bf39f55
I20260924 17:42:32.196045 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.190317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50429 (local address 127.0.98.62:45685)
0924 17:42:32.190366 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.190368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.190582 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:42:32.191489 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.191491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.191505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.191559 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:32.192645 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.193197 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.194325 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.194570 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.194714 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.195758 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:32.195759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:32.195759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:32.195765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:32.195780 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:32.195784 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:32.195844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.195930 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.195958 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:42:32.196306 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.190269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45685 (local address 127.0.98.3:50429)
0924 17:42:32.190450 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.190487 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.191292 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.191663 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.191668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.191680 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:32.192510 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.192515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.193305 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.193306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.194220 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.194222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.194726 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.194727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.194887 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.195583 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:32.195584 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:32.195615 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:32.195641 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:32.195854 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:32.195859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:32.195863 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.195935 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.196057 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260924 17:42:32.196791 18358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45685
I20260924 17:42:32.196900 18358 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:32.197094 18358 heartbeater.cc:511] Master 127.0.98.62:45685 requested a full tablet report, sending...
I20260924 17:42:32.197556 17908 ts_manager.cc:194] Registered new tserver with Master: 2066953f32bf48fe9f287343233459b2 (127.0.98.3:33975)
I20260924 17:42:32.197961 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50429
I20260924 17:42:32.203508 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:32.207958 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.204901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.0.98.62:45685)
0924 17:42:32.204952 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.204953 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.205105 (+ 152us) server_negotiation.cc:407] Connection header received
0924 17:42:32.205174 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.205175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.205186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.205256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:32.205426 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.205910 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.206665 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.206858 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.206961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.207698 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:32.207699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:32.207699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:32.207704 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:32.207718 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:32.207722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:32.207792 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.207853 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.207873 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:42:32.209400 17908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:32.210927 17908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:32.217212 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.212949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35193 (local address 127.0.0.1:47788)
0924 17:42:32.213119 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.213159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.213228 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.213831 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.213832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.213833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.213880 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.213881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.215217 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.215218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.216261 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.216263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.216930 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.217004 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217016 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.217090 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.217140 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260924 17:42:32.217226 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.213153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.0.98.2:35193)
0924 17:42:32.213685 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.213689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.213698 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:32.213742 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.213744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.213759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.213834 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.213962 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.215136 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216340 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.216830 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216927 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217016 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.217103 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.217139 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":99,"threads_started":1}
I20260924 17:42:32.217527 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.213255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.3:33975)
0924 17:42:32.213421 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.213428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.213444 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:42:32.213476 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.213477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.213489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.213556 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.213739 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.214734 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216194 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.216894 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216962 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217042 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.217402 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.217449 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260924 17:42:32.217568 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.212962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.0.1:39654)
0924 17:42:32.213357 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.213366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.213408 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.214150 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.214151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.214153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.214189 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.214191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.215426 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.215427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.216377 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.216380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.217354 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.217355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.217413 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217417 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.217466 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.217501 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":118,"threads_started":1}
I20260924 17:42:32.217331 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.212954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33975 (local address 127.0.0.1:43000)
0924 17:42:32.213336 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.213345 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.213385 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.213589 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.213592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.213594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.213659 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.213660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.214810 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.214811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.216100 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.216102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.217079 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.217081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.217143 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217147 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.217216 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.217255 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":107,"threads_started":1}
I20260924 17:42:32.217847 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.213223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.0.98.1:41653)
0924 17:42:32.213961 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.213965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.213976 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:42:32.214013 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.214015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.214030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.214107 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.214301 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.215340 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.216473 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.217175 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.217233 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.217320 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.217661 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.217724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":79,"threads_started":1}
I20260924 17:42:32.218173 18161 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 (DEFAULT_TABLE table=test-table [id=1dba86e53e0946d8b16817c10a8e1bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:32.218403 18292 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 (DEFAULT_TABLE table=test-table [id=1dba86e53e0946d8b16817c10a8e1bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:32.218549 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c637c8b574c88adcda0bca4b6e222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:32.218716 18292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c637c8b574c88adcda0bca4b6e222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:32.221468 18030 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 (DEFAULT_TABLE table=test-table [id=1dba86e53e0946d8b16817c10a8e1bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:32.221843 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c637c8b574c88adcda0bca4b6e222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:32.222123 18372 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Bootstrap starting.
I20260924 17:42:32.222307 18373 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Bootstrap starting.
I20260924 17:42:32.222820 18372 tablet_bootstrap.cc:654] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:32.222896 18373 tablet_bootstrap.cc:654] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:32.223217 18372 log.cc:824] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:32.223217 18373 log.cc:824] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:32.223984 18373 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: No bootstrap required, opened a new log
I20260924 17:42:32.224056 18373 ts_tablet_manager.cc:1397] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:42:32.224251 18372 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: No bootstrap required, opened a new log
I20260924 17:42:32.224315 18372 ts_tablet_manager.cc:1397] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:32.225165 18376 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: Bootstrap starting.
I20260924 17:42:32.225715 18376 tablet_bootstrap.cc:654] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:32.226030 18376 log.cc:824] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:32.226691 18376 tablet_bootstrap.cc:492] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: No bootstrap required, opened a new log
I20260924 17:42:32.226775 18376 ts_tablet_manager.cc:1397] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:32.226848 18372 raft_consensus.cc:348] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.226910 18373 raft_consensus.cc:348] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.227054 18373 raft_consensus.cc:374] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:32.227056 18372 raft_consensus.cc:374] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:32.227101 18372 raft_consensus.cc:729] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2b6296b4ba4e9dab0f46435a35d7ce, State: Initialized, Role: FOLLOWER
I20260924 17:42:32.227128 18373 raft_consensus.cc:729] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2066953f32bf48fe9f287343233459b2, State: Initialized, Role: FOLLOWER
I20260924 17:42:32.227257 18372 consensus_queue.cc:260] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.227272 18373 consensus_queue.cc:260] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.227519 18372 ts_tablet_manager.cc:1428] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:32.227528 18373 ts_tablet_manager.cc:1428] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 17:42:32.227612 18227 heartbeater.cc:503] Master 127.0.98.62:45685 was elected leader, sending a full tablet report...
I20260924 17:42:32.227633 18358 heartbeater.cc:503] Master 127.0.98.62:45685 was elected leader, sending a full tablet report...
I20260924 17:42:32.229430 18376 raft_consensus.cc:348] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.229586 18376 raft_consensus.cc:374] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:32.229631 18376 raft_consensus.cc:729] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbdb751c9754e7fb836e40f2a55a28c, State: Initialized, Role: FOLLOWER
I20260924 17:42:32.229748 18376 consensus_queue.cc:260] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.230026 18376 ts_tablet_manager.cc:1428] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:32.230052 18096 heartbeater.cc:503] Master 127.0.98.62:45685 was elected leader, sending a full tablet report...
I20260924 17:42:32.254797 18379 raft_consensus.cc:482] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:32.254956 18379 raft_consensus.cc:504] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.255420 18379 leader_election.cc:288] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2066953f32bf48fe9f287343233459b2 (127.0.98.3:33975), ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653)
I20260924 17:42:32.256256 18378 raft_consensus.cc:482] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:32.256357 18378 raft_consensus.cc:504] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.257140 18378 leader_election.cc:288] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e2b6296b4ba4e9dab0f46435a35d7ce (127.0.98.2:35193), ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653)
I20260924 17:42:32.259757 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.255600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33975 (local address 127.0.98.2:43085)
0924 17:42:32.255705 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.255717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.255763 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.256047 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.256048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.256050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.256131 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.256133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.257264 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.257266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.258766 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.258769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.259509 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.259578 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.259581 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.259632 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.259678 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":5}
I20260924 17:42:32.259967 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.255853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43085 (local address 127.0.98.3:33975)
0924 17:42:32.255897 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.255899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.255906 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:32.255945 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.255946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.255955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.255999 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.256227 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.257162 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.258858 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.259395 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259468 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.259477 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.259862 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.259908 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":23}
I20260924 17:42:32.260200 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2066953f32bf48fe9f287343233459b2" is_pre_election: true
I20260924 17:42:32.260306 18312 raft_consensus.cc:2454] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in term 0.
I20260924 17:42:32.260366 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.255712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34775 (local address 127.0.98.1:41653)
0924 17:42:32.255818 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.255820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.255861 (+ 41us) server_negotiation.cc:407] Connection header received
0924 17:42:32.255902 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.255904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.255918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.255998 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.256234 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.256973 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259121 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.259649 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259693 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.259702 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.260251 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.260298 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 17:42:32.260514 18117 leader_election.cc:302] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2066953f32bf48fe9f287343233459b2, 8e2b6296b4ba4e9dab0f46435a35d7ce; no voters:
I20260924 17:42:32.260646 18379 raft_consensus.cc:2781] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:32.260713 18379 raft_consensus.cc:482] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:32.260751 18379 raft_consensus.cc:3037] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:32.260906 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.255619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.98.2:34775)
0924 17:42:32.255785 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.255796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.255851 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.256037 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.256038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.256040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.256135 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.256137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.257267 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.257268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.258987 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.258989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.259767 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.260155 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.260159 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.260660 (+ 501us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.260700 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":529,"thread_start_us":105,"threads_started":1}
I20260924 17:42:32.260993 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.256802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.98.3:36325)
0924 17:42:32.256881 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.256891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.256925 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.257412 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.257413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.257415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.257477 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.257479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.258434 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.258435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.259594 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.259596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.260756 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.260757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.260827 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.260831 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.260862 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.260911 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 17:42:32.261283 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.257008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36325 (local address 127.0.98.1:41653)
0924 17:42:32.257272 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.257274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.257281 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:32.257301 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.257301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.257310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.257540 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.257571 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.258331 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.260085 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.260648 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.261152 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.261165 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.261184 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.261221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260924 17:42:32.261148 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.257770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52075 (local address 127.0.98.2:35193)
0924 17:42:32.257808 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.257810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.257818 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:42:32.257839 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.257840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.257846 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.257901 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:32.258072 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.258820 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.260278 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.260791 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.260835 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.260842 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.261066 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.261103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:42:32.261395 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.257575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35193 (local address 127.0.98.3:52075)
0924 17:42:32.257720 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.257730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:32.257781 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:32.257915 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:32.257916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:32.257922 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:32.257993 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.257994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.259022 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.259023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.260188 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:32.260191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.260900 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.260901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:32.260976 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.260980 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:32.261182 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 17:42:32.261221 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260924 17:42:32.261509 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "2066953f32bf48fe9f287343233459b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" is_pre_election: true
I20260924 17:42:32.261654 18050 raft_consensus.cc:2454] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2066953f32bf48fe9f287343233459b2 in term 0.
I20260924 17:42:32.261703 18049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" is_pre_election: true
I20260924 17:42:32.261731 18181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "2066953f32bf48fe9f287343233459b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" is_pre_election: true
I20260924 17:42:32.261798 18049 raft_consensus.cc:2454] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in term 0.
I20260924 17:42:32.261772 18379 raft_consensus.cc:504] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.261873 18181 raft_consensus.cc:2379] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2066953f32bf48fe9f287343233459b2 in current term 1: Already voted for candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in this term.
I20260924 17:42:32.261907 18379 leader_election.cc:288] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [CANDIDATE]: Term 1 election: Requested vote from peers 2066953f32bf48fe9f287343233459b2 (127.0.98.3:33975), ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653)
I20260924 17:42:32.261883 18245 leader_election.cc:302] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2066953f32bf48fe9f287343233459b2, ddbdb751c9754e7fb836e40f2a55a28c; no voters:
I20260924 17:42:32.262132 18378 raft_consensus.cc:2781] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:32.262200 18378 raft_consensus.cc:482] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:32.262200 18049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdb751c9754e7fb836e40f2a55a28c"
I20260924 17:42:32.262213 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2066953f32bf48fe9f287343233459b2"
I20260924 17:42:32.262266 18049 raft_consensus.cc:3037] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:32.262226 18378 raft_consensus.cc:3037] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:32.263154 18049 raft_consensus.cc:2454] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in term 1.
I20260924 17:42:32.263154 18378 raft_consensus.cc:504] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.263302 18378 leader_election.cc:288] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8e2b6296b4ba4e9dab0f46435a35d7ce (127.0.98.2:35193), ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653)
I20260924 17:42:32.263356 18114 leader_election.cc:302] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e2b6296b4ba4e9dab0f46435a35d7ce, ddbdb751c9754e7fb836e40f2a55a28c; no voters:
I20260924 17:42:32.263391 18312 raft_consensus.cc:2379] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in current term 1: Already voted for candidate 2066953f32bf48fe9f287343233459b2 in this term.
I20260924 17:42:32.263444 18379 raft_consensus.cc:2781] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:32.263518 18181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "2066953f32bf48fe9f287343233459b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce"
I20260924 17:42:32.263615 18379 raft_consensus.cc:686] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 LEADER]: Becoming Leader. State: Replica: 8e2b6296b4ba4e9dab0f46435a35d7ce, State: Running, Role: LEADER
I20260924 17:42:32.263578 18049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c637c8b574c88adcda0bca4b6e222" candidate_uuid: "2066953f32bf48fe9f287343233459b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdb751c9754e7fb836e40f2a55a28c"
I20260924 17:42:32.263674 18049 raft_consensus.cc:2379] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2066953f32bf48fe9f287343233459b2 in current term 1: Already voted for candidate 8e2b6296b4ba4e9dab0f46435a35d7ce in this term.
I20260924 17:42:32.263705 18379 consensus_queue.cc:237] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } }
I20260924 17:42:32.264346 18247 leader_election.cc:302] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2066953f32bf48fe9f287343233459b2; no voters: 8e2b6296b4ba4e9dab0f46435a35d7ce, ddbdb751c9754e7fb836e40f2a55a28c
I20260924 17:42:32.264451 18378 raft_consensus.cc:2726] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 17:42:32.264572 17908 catalog_manager.cc:5495] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e2b6296b4ba4e9dab0f46435a35d7ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e2b6296b4ba4e9dab0f46435a35d7ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } health_report { overall_health: UNKNOWN } } }
I20260924 17:42:32.276028 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.273774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.98.2:35193)
0924 17:42:32.273829 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:32.273831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:32.273855 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:42:32.273886 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:32.273887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:32.273903 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:32.273953 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:32.274051 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.274513 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.275290 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:32.275556 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:32.275612 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:32.275744 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:32.275839 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:32.275898 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:32.275928 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 17:42:32.275943 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:42:32.277865 18312 raft_consensus.cc:1261] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Refusing update from remote peer 8e2b6296b4ba4e9dab0f46435a35d7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:32.277890 18049 raft_consensus.cc:1261] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 1 FOLLOWER]: Refusing update from remote peer 8e2b6296b4ba4e9dab0f46435a35d7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:32.278136 18384 consensus_queue.cc:1035] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2066953f32bf48fe9f287343233459b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:32.278301 18379 consensus_queue.cc:1035] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbdb751c9754e7fb836e40f2a55a28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:32.280896 18388 mvcc.cc:205] Tried to move back new op lower bound from 7332953097327792128 to 7332953097273143296. Current Snapshot: MvccSnapshot[applied={T|T < 7332953097327792128}]
I20260924 17:42:32.282806 17908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:32.282883 17908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:32.283629 17908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92df56d574524ef49da8922aff532366: Sending DeleteTablet for 3 replicas of tablet ab3c637c8b574c88adcda0bca4b6e222
I20260924 17:42:32.283954 18292 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:32 UTC) from {username='slave'} at 127.0.0.1:43000
I20260924 17:42:32.283939 18161 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:32 UTC) from {username='slave'} at 127.0.0.1:47788
I20260924 17:42:32.284220 18398 tablet_replica.cc:330] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: stopping tablet replica
I20260924 17:42:32.284202 18030 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c637c8b574c88adcda0bca4b6e222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:32 UTC) from {username='slave'} at 127.0.0.1:39654
I20260924 17:42:32.284315 18398 raft_consensus.cc:2229] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:32.284390 18399 tablet_replica.cc:330] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c: stopping tablet replica
I20260924 17:42:32.284420 18398 raft_consensus.cc:2258] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:32.284430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17968
I20260924 17:42:32.284471 18399 raft_consensus.cc:2229] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:32.284658 18397 tablet_replica.cc:330] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: stopping tablet replica
I20260924 17:42:32.284564 18399 raft_consensus.cc:2258] T ab3c637c8b574c88adcda0bca4b6e222 P ddbdb751c9754e7fb836e40f2a55a28c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:32.284739 18397 raft_consensus.cc:2229] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:42:32.284817 18397 raft_consensus.cc:2258] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:32.285117 18398 ts_tablet_manager.cc:1905] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:32.286537 18398 ts_tablet_manager.cc:1918] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:32.286559 18397 ts_tablet_manager.cc:1905] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:32.286600 18398 log.cc:1196] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/ab3c637c8b574c88adcda0bca4b6e222
I20260924 17:42:32.286861 18398 ts_tablet_manager.cc:1939] T ab3c637c8b574c88adcda0bca4b6e222 P 2066953f32bf48fe9f287343233459b2: Deleting consensus metadata
I20260924 17:42:32.287457 17895 catalog_manager.cc:4841] TS 2066953f32bf48fe9f287343233459b2 (127.0.98.3:33975): tablet ab3c637c8b574c88adcda0bca4b6e222 (table test-table [id=1dba86e53e0946d8b16817c10a8e1bf1]) successfully deleted
I20260924 17:42:32.288394 18397 ts_tablet_manager.cc:1918] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:32.288450 18397 log.cc:1196] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/ab3c637c8b574c88adcda0bca4b6e222
I20260924 17:42:32.288635 18397 ts_tablet_manager.cc:1939] T ab3c637c8b574c88adcda0bca4b6e222 P 8e2b6296b4ba4e9dab0f46435a35d7ce: Deleting consensus metadata
I20260924 17:42:32.288976 17894 catalog_manager.cc:4841] TS 8e2b6296b4ba4e9dab0f46435a35d7ce (127.0.98.2:35193): tablet ab3c637c8b574c88adcda0bca4b6e222 (table test-table [id=1dba86e53e0946d8b16817c10a8e1bf1]) successfully deleted
W20260924 17:42:32.293299 17892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41653 (error 108)
I20260924 17:42:32.293619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18099
I20260924 17:42:32.293743 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:32.293604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.0.1:39660)
0924 17:42:32.293662 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:32.293706 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 17:42:32.293854 17892 catalog_manager.cc:4584] TS ddbdb751c9754e7fb836e40f2a55a28c (127.0.98.1:41653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab3c637c8b574c88adcda0bca4b6e222: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41653: connect: Connection refused (error 111)
I20260924 17:42:32.297958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18230
I20260924 17:42:32.302621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 17877
2026-09-24T17:42:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (657 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:32 dist-test-slave-d0pf krb5kdc[18403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:32 dist-test-slave-d0pf krb5kdc[18403](info): set up 2 sockets
Sep 24 17:42:32 dist-test-slave-d0pf krb5kdc[18403](info): commencing operation
krb5kdc: starting...
W20260924 17:42:33.338781 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:33.456248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:33.536432 18416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:33.536646 18416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:33.536669 18416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:33.537843 18416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:33.537886 18416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:33.537900 18416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:33.537914 18416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:33.537925 18416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:33.539455 18416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:33.539712 18416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:33.539960 18416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:33.544737 18424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:33.544721 18422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:33.544721 18421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:33.544829 18416 server_base.cc:1034] running on GCE node
I20260924 17:42:33.545192 18416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:33.545481 18416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:33.546633 18416 hybrid_clock.cc:648] HybridClock initialized: now 1790271753546607 us; error 47 us; skew 500 ppm
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:33.548981 18416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:33.549425 18416 webserver.cc:460] Webserver started at http://127.0.98.62:46125/ using document root <none> and password file <none>
I20260924 17:42:33.549615 18416 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:33.549656 18416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:33.549742 18416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:33.550734 18416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "dfc8f113c49a4c31a9533f3daa450daf"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "466077ba6e191d8847f12cc0fbb13f3e"
server_key_iv: "73f9611b11982101c948154cccab9cbe"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.551025 18416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfc8f113c49a4c31a9533f3daa450daf"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "466077ba6e191d8847f12cc0fbb13f3e"
server_key_iv: "73f9611b11982101c948154cccab9cbe"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.552374 18416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:33.553100 18431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.553303 18416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:33.553391 18416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "dfc8f113c49a4c31a9533f3daa450daf"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "466077ba6e191d8847f12cc0fbb13f3e"
server_key_iv: "73f9611b11982101c948154cccab9cbe"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.553444 18416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:33.562760 18416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:33.565030 18416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:33.565140 18416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:33.568871 18416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34697
I20260924 17:42:33.568923 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34697 every 8 connection(s)
I20260924 17:42:33.569280 18416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:33.570171 18484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:33.570801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18416
I20260924 17:42:33.570920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:33.571152 392 external_mini_cluster.cc:1428] Setting key 6c4a5d90443337a26ddb06ead19b1514
I20260924 17:42:33.573277 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: Bootstrap starting.
I20260924 17:42:33.573974 18484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:33.574290 18484 log.cc:824] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:33.574977 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: No bootstrap required, opened a new log
I20260924 17:42:33.577107 18484 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260924 17:42:33.577237 18484 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:33.577258 18484 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc8f113c49a4c31a9533f3daa450daf, State: Initialized, Role: FOLLOWER
I20260924 17:42:33.577390 18484 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260924 17:42:33.577471 18484 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:33.577492 18484 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:33.577564 18484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:33.578308 18484 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:33.578418 18484 leader_election.cc:302] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc8f113c49a4c31a9533f3daa450daf; no voters:
I20260924 17:42:33.578611 18484 leader_election.cc:288] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:33.578639 18489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:33.578752 18489 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [term 1 LEADER]: Becoming Leader. State: Replica: dfc8f113c49a4c31a9533f3daa450daf, State: Running, Role: LEADER
I20260924 17:42:33.578877 18484 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:33.578891 18489 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260924 17:42:33.579290 18489 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc8f113c49a4c31a9533f3daa450daf. Latest consensus state: current_term: 1 leader_uuid: "dfc8f113c49a4c31a9533f3daa450daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } } }
I20260924 17:42:33.579329 18490 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc8f113c49a4c31a9533f3daa450daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc8f113c49a4c31a9533f3daa450daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } } }
I20260924 17:42:33.579371 18489 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:33.579403 18490 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf [sys.catalog]: This master's current role is: LEADER
W20260924 17:42:33.580688 18505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:33.580754 18505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:33.580801 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:33.580964 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:33.582608 18495 catalog_manager.cc:1347] Generated new cluster ID: e5f65954a7bd47a6bbbfa3fe47c74fc7
I20260924 17:42:33.582645 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:33.571674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.98.62:34697)
0924 17:42:33.571935 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:33.571945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:33.571976 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:42:33.572543 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:33.572559 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:33.572625 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:33.572772 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:33.573367 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.574722 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.575582 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.576200 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.576316 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:33.578969 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:33.578973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:33.578973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:33.579009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:33.581061 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.581422 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.581425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.581426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.581551 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.581791 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.581792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.581793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.582053 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:33.582209 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:33.582327 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:42:33.582461 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":60,"threads_started":1}
I20260924 17:42:33.582662 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:33.597224 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:33.597795 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:33.600881 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: Generated new TSK 0
I20260924 17:42:33.601078 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:42:33.629843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:33.714901 18510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:33.715170 18510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:33.715198 18510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:33.716439 18510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:33.716483 18510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:33.716511 18510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:33.717934 18510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:33.718123 18510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:33.718293 18510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:33.723229 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:33.723248 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:33.723268 18510 server_base.cc:1034] running on GCE node
W20260924 17:42:33.723248 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:33.723661 18510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:33.723914 18510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:33.725056 18510 hybrid_clock.cc:648] HybridClock initialized: now 1790271753725033 us; error 41 us; skew 500 ppm
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:33.727593 18510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:42:33.728029 18510 webserver.cc:460] Webserver started at http://127.0.98.1:44213/ using document root <none> and password file <none>
I20260924 17:42:33.728252 18510 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:33.728317 18510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:33.728464 18510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:33.729519 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "13c4c304775b49c9b0687762cf66cc59"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "2ca20a241e50000449cc1f7bf094a8f3"
server_key_iv: "0766625acc0c40a76c98907e732f570e"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.729889 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13c4c304775b49c9b0687762cf66cc59"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "2ca20a241e50000449cc1f7bf094a8f3"
server_key_iv: "0766625acc0c40a76c98907e732f570e"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.731277 18510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:42:33.732075 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.732242 18510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:33.732344 18510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "13c4c304775b49c9b0687762cf66cc59"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "2ca20a241e50000449cc1f7bf094a8f3"
server_key_iv: "0766625acc0c40a76c98907e732f570e"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.732430 18510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:33.746662 18510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:33.748557 18510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:33.748720 18510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:33.748963 18510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:33.749265 18510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:33.749317 18510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.749358 18510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:33.749405 18510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.755208 18510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43007
I20260924 17:42:33.755283 18638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43007 every 8 connection(s)
I20260924 17:42:33.755623 18510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:33.764923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18510
I20260924 17:42:33.765080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:33.765321 392 external_mini_cluster.cc:1428] Setting key 0688200e347a2a2e63e63551dabe82d9
I20260924 17:42:33.765430 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:33.756418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49641 (local address 127.0.98.62:34697)
0924 17:42:33.756497 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:33.756499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:33.756662 (+ 163us) server_negotiation.cc:407] Connection header received
0924 17:42:33.757740 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:33.757741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:33.757753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:33.757811 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:33.758406 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.758920 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.760016 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.760229 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.760286 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:33.762380 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:33.762382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:33.762382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:33.762389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:33.763840 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.764200 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.764200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.764201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.764252 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.764464 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.764465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.764465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.764575 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:33.764625 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:33.765247 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 17:42:33.765321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:42:33.765669 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:33.756370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.1:49641)
0924 17:42:33.756538 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:33.756576 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:33.757565 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:33.757870 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:33.757873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:33.758137 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:33.758331 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:33.758334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.759027 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.759028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:33.759932 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:33.759935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.760309 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.760310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:33.760414 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:33.760906 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:33.760914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:33.762300 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:33.763909 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:33.763914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:33.763916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:33.764137 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:33.764307 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:33.764307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:33.764307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:33.764412 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:33.764754 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:33.764760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:33.764935 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 17:42:33.765204 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 17:42:33.765393 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260924 17:42:33.766211 18639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260924 17:42:33.766337 18639 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:33.766520 18639 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260924 17:42:33.767096 18448 ts_manager.cc:194] Registered new tserver with Master: 13c4c304775b49c9b0687762cf66cc59 (127.0.98.1:43007)
I20260924 17:42:33.768044 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49641
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:42:33.790344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:33.870651 18644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:33.870863 18644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:33.870886 18644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:33.872085 18644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:33.872131 18644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:33.872159 18644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:33.873657 18644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:33.873898 18644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:33.874140 18644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:33.879159 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:33.879129 18649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:33.879194 18650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:33.879351 18644 server_base.cc:1034] running on GCE node
I20260924 17:42:33.879645 18644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:33.879931 18644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:33.881100 18644 hybrid_clock.cc:648] HybridClock initialized: now 1790271753881086 us; error 28 us; skew 500 ppm
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:33.883756 18644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:42:33.884208 18644 webserver.cc:460] Webserver started at http://127.0.98.2:35223/ using document root <none> and password file <none>
I20260924 17:42:33.884409 18644 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:33.884452 18644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:33.884541 18644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:33.885571 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "933da04a015e49dcb4ccbd2f19a55eee"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "6d2fe477f1029e4bbaaa005cce84301a"
server_key_iv: "1461b1be07d4bf6e4be898c2af045a65"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.885871 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "933da04a015e49dcb4ccbd2f19a55eee"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "6d2fe477f1029e4bbaaa005cce84301a"
server_key_iv: "1461b1be07d4bf6e4be898c2af045a65"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.887171 18644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:42:33.887926 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.888160 18644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:33.888262 18644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "933da04a015e49dcb4ccbd2f19a55eee"
format_stamp: "Formatted at 2026-09-24 17:42:33 on dist-test-slave-d0pf"
server_key: "6d2fe477f1029e4bbaaa005cce84301a"
server_key_iv: "1461b1be07d4bf6e4be898c2af045a65"
server_key_version: "encryptionkey@0"
I20260924 17:42:33.888322 18644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:33.908337 18644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:33.910442 18644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:33.910688 18644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:33.910930 18644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:33.911326 18644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:33.911366 18644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.911417 18644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:33.911445 18644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:33.917169 18644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36101
I20260924 17:42:33.917264 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36101 every 8 connection(s)
I20260924 17:42:33.917585 18644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:42:33 dist-test-slave-d0pf krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:33.925403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18644
I20260924 17:42:33.925531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:33.925765 392 external_mini_cluster.cc:1428] Setting key 4705ce5ddb28b46190802a76e4ae1a30
I20260924 17:42:33.927574 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:33.918381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37043 (local address 127.0.98.62:34697)
0924 17:42:33.918468 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:33.918470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:33.918664 (+ 194us) server_negotiation.cc:407] Connection header received
0924 17:42:33.919664 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:33.919666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:33.919680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:33.919737 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:33.920382 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.920880 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.921901 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.922107 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.922193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:33.924418 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:33.924420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:33.924421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:33.924430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:33.926018 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.926420 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.926421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.926421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.926507 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:33.926785 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:33.926786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:33.926786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:33.926941 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:33.927014 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:33.927332 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 17:42:33.927427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:42:33.927600 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:33.918369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.2:37043)
0924 17:42:33.918533 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:33.918574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:33.919538 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:33.919815 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:33.919819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:33.920087 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:33.920283 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:33.920287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.920975 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.920976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:33.921799 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:33.921801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:33.922204 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:33.922205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:33.922335 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:33.922882 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:33.922891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:33.924299 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:33.926112 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:33.926119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:33.926121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:33.926340 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:33.926584 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:33.926595 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:33.926595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:33.926714 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:33.927008 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:33.927013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:33.927193 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 17:42:33.927284 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:42:33.927416 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260924 17:42:33.928398 18773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260924 17:42:33.928514 18773 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:33.928715 18773 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260924 17:42:33.929070 18448 ts_manager.cc:194] Registered new tserver with Master: 933da04a015e49dcb4ccbd2f19a55eee (127.0.98.2:36101)
I20260924 17:42:33.929474 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37043
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:42:33.953231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:34.034911 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:34.035214 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:34.035257 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:34.036490 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:34.036559 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:34.036605 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:34.038126 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:34.038389 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:34.038632 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:34.043627 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.043622 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.043833 18778 server_base.cc:1034] running on GCE node
W20260924 17:42:34.043619 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.044140 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:34.044399 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:34.045564 18778 hybrid_clock.cc:648] HybridClock initialized: now 1790271754045549 us; error 31 us; skew 500 ppm
Sep 24 17:42:34 dist-test-slave-d0pf krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:34.048122 18778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:42:34.048616 18778 webserver.cc:460] Webserver started at http://127.0.98.3:45417/ using document root <none> and password file <none>
I20260924 17:42:34.048842 18778 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:34.048892 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:34.049072 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:34.050145 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e28b76b6c974c91bf49b8d9bb052a2c"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "2b7ba4a18491433c4919f52f1fc00fb1"
server_key_iv: "7452bf89c9f7703714f95d270e9be686"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.050513 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e28b76b6c974c91bf49b8d9bb052a2c"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "2b7ba4a18491433c4919f52f1fc00fb1"
server_key_iv: "7452bf89c9f7703714f95d270e9be686"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.051911 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:34.052680 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.052868 18778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:34.052954 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "1e28b76b6c974c91bf49b8d9bb052a2c"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "2b7ba4a18491433c4919f52f1fc00fb1"
server_key_iv: "7452bf89c9f7703714f95d270e9be686"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.053045 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:34.061043 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:34.063165 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:34.063288 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:34.063477 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:34.063788 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:34.063822 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.063876 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:34.063905 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.069710 18778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44763
I20260924 17:42:34.069787 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44763 every 8 connection(s)
I20260924 17:42:34.070120 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:42:34 dist-test-slave-d0pf krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:34.078248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18778
I20260924 17:42:34.078377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:34.078660 392 external_mini_cluster.cc:1428] Setting key 01518e8baebb69166333df0535ea259b
I20260924 17:42:34.080708 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.070956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44565 (local address 127.0.98.62:34697)
0924 17:42:34.071022 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.071025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.071263 (+ 238us) server_negotiation.cc:407] Connection header received
0924 17:42:34.072251 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.072253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.072267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.072318 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.072986 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.073495 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.074521 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.074761 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.074831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.077330 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.077332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.077332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:34.077343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.079128 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:34.079681 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:34.079682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:34.079682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:34.079755 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:34.079993 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:34.079994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:34.079994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:34.080103 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.080187 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.080512 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.080616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:42:34.080880 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.070922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.3:44565)
0924 17:42:34.071142 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.071179 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.072127 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.072396 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.072400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.072658 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:34.072856 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.072861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.073598 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.073598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.074424 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.074427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.074871 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.074872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.075003 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.075648 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:34.075656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:34.077222 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:34.079222 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:34.079230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:34.079233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:34.079574 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:34.079819 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:34.079820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:34.079820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:34.079934 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:34.080185 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:34.080191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:34.080338 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.080469 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.080639 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":146,"threads_started":1}
I20260924 17:42:34.081489 18907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260924 17:42:34.081585 18907 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:34.081732 18907 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260924 17:42:34.082064 18448 ts_manager.cc:194] Registered new tserver with Master: 1e28b76b6c974c91bf49b8d9bb052a2c (127.0.98.3:44763)
I20260924 17:42:34.082468 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44565
I20260924 17:42:34.090613 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:34.098724 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.092065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.98.62:34697)
0924 17:42:34.092127 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.092129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.092243 (+ 114us) server_negotiation.cc:407] Connection header received
0924 17:42:34.092321 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.092322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.092338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.092402 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.092992 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.093500 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.094260 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.094460 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.094565 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.095978 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.095979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.095980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:34.095988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.097294 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:34.097817 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:34.097818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:34.097819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:34.097882 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:34.098112 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:34.098113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:34.098114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:34.098260 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.098378 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.098553 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.098619 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:42:34.100490 18448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:34.102388 18448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:34.109257 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.104943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36101 (local address 127.0.0.1:59874)
0924 17:42:34.105253 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.105263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.105307 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.105543 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.105546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.105547 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.105614 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.105615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.106953 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.106954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.108098 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.108100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.108940 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.108941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.109024 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.109043 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.109110 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.109165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":39,"thread_start_us":90,"threads_started":1}
I20260924 17:42:34.109386 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.104935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44763 (local address 127.0.0.1:59116)
0924 17:42:34.105089 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.105131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.105199 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.105919 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.105920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.105922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.105966 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.105968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.106907 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.106908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.108265 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.108268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.109065 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.109066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.109129 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.109134 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.109200 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.109244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":38,"thread_start_us":46,"threads_started":1}
I20260924 17:42:34.109676 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.105184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.2:36101)
0924 17:42:34.105372 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.105377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.105389 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:42:34.105429 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.105430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.105443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.105516 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.105698 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.106833 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.108192 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.108843 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.108919 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.108977 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.109551 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.109601 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260924 17:42:34.109688 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.104943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43007 (local address 127.0.0.1:36226)
0924 17:42:34.105304 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.105312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.105348 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.106156 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.106157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.106159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.106193 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.106194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.107361 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.107362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.108430 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.108431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.109451 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.109452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.109518 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.109521 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.109592 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.109631 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":12,"thread_start_us":91,"threads_started":1}
I20260924 17:42:34.109732 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.105185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.0.98.1:43007)
0924 17:42:34.106006 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.106009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.106018 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:34.106052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.106054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.106068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.106144 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.106252 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.107262 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.108529 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.109305 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.109366 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.109417 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.109610 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.109649 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":92,"threads_started":1}
I20260924 17:42:34.110045 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.105084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.98.3:44763)
0924 17:42:34.105681 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.105686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.105695 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:34.105731 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.105733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.105751 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.105861 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.106028 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.106803 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.108372 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.108983 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.109062 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.109119 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.109213 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.109251 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":94,"threads_started":1}
I20260924 17:42:34.110368 18707 tablet_service.cc:1467] Processing CreateTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 (DEFAULT_TABLE table=test-table [id=5d082adc86594be2b6a94f034fbdd671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.110395 18573 tablet_service.cc:1467] Processing CreateTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 (DEFAULT_TABLE table=test-table [id=5d082adc86594be2b6a94f034fbdd671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.110759 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7832327f26794dfcadecb9ec2abd1fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.110759 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7832327f26794dfcadecb9ec2abd1fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.110786 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 (DEFAULT_TABLE table=test-table [id=5d082adc86594be2b6a94f034fbdd671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.111114 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7832327f26794dfcadecb9ec2abd1fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.113337 18924 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: Bootstrap starting.
I20260924 17:42:34.114033 18924 tablet_bootstrap.cc:654] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.114401 18926 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Bootstrap starting.
I20260924 17:42:34.114650 18924 log.cc:824] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.115084 18926 tablet_bootstrap.cc:654] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.115224 18925 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Bootstrap starting.
I20260924 17:42:34.115527 18924 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: No bootstrap required, opened a new log
I20260924 17:42:34.115613 18924 ts_tablet_manager.cc:1397] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:34.115640 18926 log.cc:824] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.115825 18925 tablet_bootstrap.cc:654] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.116367 18925 log.cc:824] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.116439 18926 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: No bootstrap required, opened a new log
I20260924 17:42:34.116535 18926 ts_tablet_manager.cc:1397] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:34.117797 18924 raft_consensus.cc:348] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.118005 18924 raft_consensus.cc:374] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.118042 18924 raft_consensus.cc:729] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c4c304775b49c9b0687762cf66cc59, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.118161 18924 consensus_queue.cc:260] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.118481 18924 ts_tablet_manager.cc:1428] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:34.118500 18639 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
I20260924 17:42:34.118628 18926 raft_consensus.cc:348] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.118773 18926 raft_consensus.cc:374] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.118882 18926 raft_consensus.cc:729] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e28b76b6c974c91bf49b8d9bb052a2c, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.119097 18926 consensus_queue.cc:260] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.119385 18926 ts_tablet_manager.cc:1428] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 17:42:34.119438 18907 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
I20260924 17:42:34.120173 18925 tablet_bootstrap.cc:492] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: No bootstrap required, opened a new log
I20260924 17:42:34.120253 18925 ts_tablet_manager.cc:1397] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 17:42:34.121917 18925 raft_consensus.cc:348] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.122108 18925 raft_consensus.cc:374] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.122166 18925 raft_consensus.cc:729] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933da04a015e49dcb4ccbd2f19a55eee, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.122292 18925 consensus_queue.cc:260] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.122606 18925 ts_tablet_manager.cc:1428] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:42:34.122609 18773 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
W20260924 17:42:34.168187 18774 tablet.cc:2369] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:42:34.171717 18930 raft_consensus.cc:482] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:34.171832 18930 raft_consensus.cc:504] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.172178 18930 leader_election.cc:288] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e28b76b6c974c91bf49b8d9bb052a2c (127.0.98.3:44763), 933da04a015e49dcb4ccbd2f19a55eee (127.0.98.2:36101)
I20260924 17:42:34.175071 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.172376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36101 (local address 127.0.98.1:60467)
0924 17:42:34.172432 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.172444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.172501 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.172733 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.172733 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.172735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.172811 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.172812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.173456 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.173457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.174228 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.174230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.174789 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.174871 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.174874 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.174936 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.174979 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 17:42:34.175091 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.172546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60467 (local address 127.0.98.2:36101)
0924 17:42:34.172621 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.172623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.172630 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:34.172652 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.172653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.172663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.172726 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.172883 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.173369 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174326 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.174704 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174787 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.174798 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.174956 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.175001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:42:34.175236 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.172444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52797 (local address 127.0.98.3:44763)
0924 17:42:34.172500 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.172503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.172770 (+ 267us) server_negotiation.cc:407] Connection header received
0924 17:42:34.172851 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.172852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.172861 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.172931 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.173089 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.173551 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174485 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.174861 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174942 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.174951 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.175137 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.175175 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:42:34.175244 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.172384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44763 (local address 127.0.98.1:52797)
0924 17:42:34.172686 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.172695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.172777 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.172937 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.172938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.172939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.173016 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.173017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.173637 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.173638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.174391 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.174393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.174946 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.174947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.175069 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.175074 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.175120 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.175159 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":98,"thread_start_us":115,"threads_started":1}
I20260924 17:42:34.175508 18727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7832327f26794dfcadecb9ec2abd1fe2" candidate_uuid: "13c4c304775b49c9b0687762cf66cc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933da04a015e49dcb4ccbd2f19a55eee" is_pre_election: true
I20260924 17:42:34.175591 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7832327f26794dfcadecb9ec2abd1fe2" candidate_uuid: "13c4c304775b49c9b0687762cf66cc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" is_pre_election: true
I20260924 17:42:34.175678 18727 raft_consensus.cc:2454] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c4c304775b49c9b0687762cf66cc59 in term 0.
I20260924 17:42:34.175716 18861 raft_consensus.cc:2454] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c4c304775b49c9b0687762cf66cc59 in term 0.
I20260924 17:42:34.175915 18529 leader_election.cc:302] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c4c304775b49c9b0687762cf66cc59, 933da04a015e49dcb4ccbd2f19a55eee; no voters:
I20260924 17:42:34.176061 18930 raft_consensus.cc:2781] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:34.176118 18930 raft_consensus.cc:482] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:34.176172 18930 raft_consensus.cc:3037] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.176854 18930 raft_consensus.cc:504] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.176995 18930 leader_election.cc:288] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [CANDIDATE]: Term 1 election: Requested vote from peers 1e28b76b6c974c91bf49b8d9bb052a2c (127.0.98.3:44763), 933da04a015e49dcb4ccbd2f19a55eee (127.0.98.2:36101)
I20260924 17:42:34.177129 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7832327f26794dfcadecb9ec2abd1fe2" candidate_uuid: "13c4c304775b49c9b0687762cf66cc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c"
I20260924 17:42:34.177181 18727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7832327f26794dfcadecb9ec2abd1fe2" candidate_uuid: "13c4c304775b49c9b0687762cf66cc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933da04a015e49dcb4ccbd2f19a55eee"
I20260924 17:42:34.177227 18861 raft_consensus.cc:3037] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.177287 18727 raft_consensus.cc:3037] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.177959 18861 raft_consensus.cc:2454] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c4c304775b49c9b0687762cf66cc59 in term 1.
I20260924 17:42:34.178045 18727 raft_consensus.cc:2454] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c4c304775b49c9b0687762cf66cc59 in term 1.
I20260924 17:42:34.178187 18527 leader_election.cc:302] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c4c304775b49c9b0687762cf66cc59, 1e28b76b6c974c91bf49b8d9bb052a2c; no voters:
I20260924 17:42:34.178272 18930 raft_consensus.cc:2781] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:34.178503 18930 raft_consensus.cc:686] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [term 1 LEADER]: Becoming Leader. State: Replica: 13c4c304775b49c9b0687762cf66cc59, State: Running, Role: LEADER
I20260924 17:42:34.178613 18930 consensus_queue.cc:237] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } }
I20260924 17:42:34.179495 18448 catalog_manager.cc:5495] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c4c304775b49c9b0687762cf66cc59 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "13c4c304775b49c9b0687762cf66cc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13c4c304775b49c9b0687762cf66cc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 } health_report { overall_health: UNKNOWN } } }
I20260924 17:42:34.203135 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.200856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.0.98.1:43007)
0924 17:42:34.200924 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.200926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.200934 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:42:34.200987 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.200988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.201010 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.201057 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:34.201165 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.201628 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.202353 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.202535 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.202584 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.202705 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:34.202791 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:34.202858 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.202964 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.202978 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:42:34.205090 18861 raft_consensus.cc:1261] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 1 FOLLOWER]: Refusing update from remote peer 13c4c304775b49c9b0687762cf66cc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:34.205094 18727 raft_consensus.cc:1261] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 1 FOLLOWER]: Refusing update from remote peer 13c4c304775b49c9b0687762cf66cc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:34.205482 18930 consensus_queue.cc:1035] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e28b76b6c974c91bf49b8d9bb052a2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:34.205614 18934 consensus_queue.cc:1035] T 7832327f26794dfcadecb9ec2abd1fe2 P 13c4c304775b49c9b0687762cf66cc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "933da04a015e49dcb4ccbd2f19a55eee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:34.208105 18937 mvcc.cc:205] Tried to move back new op lower bound from 7332953105221648384 to 7332953105116917760. Current Snapshot: MvccSnapshot[applied={T|T < 7332953105221648384}]
I20260924 17:42:34.208784 18939 mvcc.cc:205] Tried to move back new op lower bound from 7332953105221648384 to 7332953105116917760. Current Snapshot: MvccSnapshot[applied={T|T < 7332953105221648384 or (T in {7332953105221648384})}]
I20260924 17:42:34.210618 18447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:34.210738 18447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:34.211601 18447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfc8f113c49a4c31a9533f3daa450daf: Sending DeleteTablet for 3 replicas of tablet 7832327f26794dfcadecb9ec2abd1fe2
I20260924 17:42:34.211948 18573 tablet_service.cc:1514] Processing DeleteTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36226
I20260924 17:42:34.211946 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59116
I20260924 17:42:34.212209 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet 7832327f26794dfcadecb9ec2abd1fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59874
I20260924 17:42:34.212272 18947 tablet_replica.cc:330] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: stopping tablet replica
I20260924 17:42:34.212371 18947 raft_consensus.cc:2229] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:34.212472 18949 tablet_replica.cc:330] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: stopping tablet replica
I20260924 17:42:34.212551 18947 raft_consensus.cc:2258] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:34.212553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18510
I20260924 17:42:34.213292 18949 raft_consensus.cc:2229] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:34.213431 18949 raft_consensus.cc:2258] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:34.213656 18947 ts_tablet_manager.cc:1905] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:34.213907 18949 ts_tablet_manager.cc:1905] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:34.215575 18947 ts_tablet_manager.cc:1918] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:34.215655 18947 log.cc:1196] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/7832327f26794dfcadecb9ec2abd1fe2
I20260924 17:42:34.215806 18949 ts_tablet_manager.cc:1918] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:34.215916 18949 log.cc:1196] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/7832327f26794dfcadecb9ec2abd1fe2
I20260924 17:42:34.215932 18947 ts_tablet_manager.cc:1939] T 7832327f26794dfcadecb9ec2abd1fe2 P 1e28b76b6c974c91bf49b8d9bb052a2c: Deleting consensus metadata
I20260924 17:42:34.216205 18949 ts_tablet_manager.cc:1939] T 7832327f26794dfcadecb9ec2abd1fe2 P 933da04a015e49dcb4ccbd2f19a55eee: Deleting consensus metadata
I20260924 17:42:34.216389 18433 catalog_manager.cc:4841] TS 1e28b76b6c974c91bf49b8d9bb052a2c (127.0.98.3:44763): tablet 7832327f26794dfcadecb9ec2abd1fe2 (table test-table [id=5d082adc86594be2b6a94f034fbdd671]) successfully deleted
I20260924 17:42:34.216787 18435 catalog_manager.cc:4841] TS 933da04a015e49dcb4ccbd2f19a55eee (127.0.98.2:36101): tablet 7832327f26794dfcadecb9ec2abd1fe2 (table test-table [id=5d082adc86594be2b6a94f034fbdd671]) successfully deleted
W20260924 17:42:34.218348 18432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43007 (error 108)
I20260924 17:42:34.218560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18644
I20260924 17:42:34.222468 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.222309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43007 (local address 127.0.0.1:36246)
0924 17:42:34.222386 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.222433 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 17:42:34.222613 18432 catalog_manager.cc:4584] TS 13c4c304775b49c9b0687762cf66cc59 (127.0.98.1:43007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7832327f26794dfcadecb9ec2abd1fe2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43007: connect: Connection refused (error 111)
I20260924 17:42:34.223754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18778
I20260924 17:42:34.228956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18416
2026-09-24T17:42:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T17:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:34Z Disabled control of system clock
I20260924 17:42:34.257483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34033
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:34.343817 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:34.344054 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:34.344076 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:34.345316 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:34.345371 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:34.345384 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:34.345397 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:34.345409 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:34.346817 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:34.347117 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:34.347430 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:34.352293 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.352329 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.352268 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.352442 18954 server_base.cc:1034] running on GCE node
I20260924 17:42:34.352772 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:34.353081 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:34.354218 18954 hybrid_clock.cc:648] HybridClock initialized: now 1790271754354209 us; error 30 us; skew 500 ppm
I20260924 17:42:34.355476 18954 webserver.cc:460] Webserver started at http://127.0.98.62:32773/ using document root <none> and password file <none>
I20260924 17:42:34.355738 18954 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:34.355784 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:34.355952 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:34.357043 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "582488a2974d4bb1ae07cac66fc90d54"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "456b68d06e6590c83853c74f28b010e4"
server_key_iv: "b6d67994fbec5e91a503e5ddeaa693cc"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.357417 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "582488a2974d4bb1ae07cac66fc90d54"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "456b68d06e6590c83853c74f28b010e4"
server_key_iv: "b6d67994fbec5e91a503e5ddeaa693cc"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.358769 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:34.359642 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.359853 18954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:42:34.359958 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "582488a2974d4bb1ae07cac66fc90d54"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "456b68d06e6590c83853c74f28b010e4"
server_key_iv: "b6d67994fbec5e91a503e5ddeaa693cc"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.360045 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:34.372439 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:34.372771 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:34.372931 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:34.377264 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34033
I20260924 17:42:34.377300 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34033 every 8 connection(s)
I20260924 17:42:34.377656 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:34.378714 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.381083 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: Bootstrap starting.
I20260924 17:42:34.381628 19021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.381886 19021 log.cc:824] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.382376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 18954
I20260924 17:42:34.382506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:34.382683 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: No bootstrap required, opened a new log
I20260924 17:42:34.382691 392 external_mini_cluster.cc:1428] Setting key 6f4142fa444fbae21279ed65029a3ace
I20260924 17:42:34.385339 19021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } }
I20260924 17:42:34.385525 19021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.385568 19021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582488a2974d4bb1ae07cac66fc90d54, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.385697 19021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } }
I20260924 17:42:34.385779 19021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:34.385808 19021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:34.385854 19021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.386559 19021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } }
I20260924 17:42:34.386696 19021 leader_election.cc:302] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 582488a2974d4bb1ae07cac66fc90d54; no voters:
I20260924 17:42:34.386868 19021 leader_election.cc:288] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:34.386899 19026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:34.387009 19026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [term 1 LEADER]: Becoming Leader. State: Replica: 582488a2974d4bb1ae07cac66fc90d54, State: Running, Role: LEADER
I20260924 17:42:34.387169 19026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } }
I20260924 17:42:34.387167 19021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:34.388599 19028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582488a2974d4bb1ae07cac66fc90d54. Latest consensus state: current_term: 1 leader_uuid: "582488a2974d4bb1ae07cac66fc90d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } } }
I20260924 17:42:34.388600 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "582488a2974d4bb1ae07cac66fc90d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582488a2974d4bb1ae07cac66fc90d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34033 } } }
I20260924 17:42:34.388862 19028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:34.388875 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:34.389016 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.383274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.0.98.62:34033)
0924 17:42:34.383501 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.383509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.383542 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:42:34.384057 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.384064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.384120 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.384251 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.384493 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.385389 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.386526 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.387195 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.387663 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.388045 (+ 382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.388049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.388049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:34.388254 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.388320 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:34.388337 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.388410 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.388691 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.388775 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
W20260924 17:42:34.389220 19040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:34.389262 19040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:34.389564 19041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:34.389750 19041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:34.391223 19041 catalog_manager.cc:1347] Generated new cluster ID: cc512bea9ccb4de3a2171581ede4f1cf
I20260924 17:42:34.391285 19041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:34.400635 19041 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:34.401345 19041 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:34.405354 19041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: Generated new TSK 0
I20260924 17:42:34.405550 19041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:34.406813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:34.488426 19045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:34.488718 19045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:34.488747 19045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:34.490120 19045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:34.490173 19045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:34.490209 19045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:34.491788 19045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:34.492044 19045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:34.492384 19045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:34.498308 19050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.498304 19051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.498598 19045 server_base.cc:1034] running on GCE node
W20260924 17:42:34.498742 19053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.498932 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:34.499249 19045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:34.500409 19045 hybrid_clock.cc:648] HybridClock initialized: now 1790271754500389 us; error 37 us; skew 500 ppm
I20260924 17:42:34.501765 19045 webserver.cc:460] Webserver started at http://127.0.98.1:43501/ using document root <none> and password file <none>
I20260924 17:42:34.502013 19045 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:34.502059 19045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:34.502219 19045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:34.503398 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba9f35210e774d0faa7f2677fb5be60a"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "680a8d860cb912ef90c6cfc3fb240648"
server_key_iv: "532c49bfac314366070344e08b5caca9"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.503762 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9f35210e774d0faa7f2677fb5be60a"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "680a8d860cb912ef90c6cfc3fb240648"
server_key_iv: "532c49bfac314366070344e08b5caca9"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.505147 19045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:34.505966 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.506174 19045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:34.506278 19045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "ba9f35210e774d0faa7f2677fb5be60a"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "680a8d860cb912ef90c6cfc3fb240648"
server_key_iv: "532c49bfac314366070344e08b5caca9"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.506374 19045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:34.518836 19045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:34.519253 19045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:34.519424 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:34.519675 19045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:34.520017 19045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:34.520074 19045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.520119 19045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:34.520149 19045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.525766 19045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37803
I20260924 17:42:34.525815 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37803 every 8 connection(s)
I20260924 17:42:34.526283 19045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:34.531912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19045
I20260924 17:42:34.532051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:34.532096 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.527211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50441 (local address 127.0.98.62:34033)
0924 17:42:34.527287 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.527290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.527591 (+ 301us) server_negotiation.cc:407] Connection header received
0924 17:42:34.528072 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.528073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.528088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.528143 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.528837 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.529364 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.530335 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.530540 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.530610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.531740 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.531742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.531742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:34.531750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.531785 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:34.531790 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.531860 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.531958 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.531992 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:42:34.532318 392 external_mini_cluster.cc:1428] Setting key 4220a7ac269338c5baece5e9d10e2c62
I20260924 17:42:34.532389 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.527199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34033 (local address 127.0.98.1:50441)
0924 17:42:34.527469 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.527505 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.527991 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.528196 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.528208 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.528217 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:34.528726 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.528730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.529453 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.529454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.530240 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.530242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.530627 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.530628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.530749 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.531532 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:34.531532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:34.531563 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:34.531588 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:34.531869 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:34.531874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:34.531877 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.531938 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.532075 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":137,"threads_started":1}
I20260924 17:42:34.533123 19173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34033
I20260924 17:42:34.533248 19173 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:34.533463 19173 heartbeater.cc:511] Master 127.0.98.62:34033 requested a full tablet report, sending...
I20260924 17:42:34.533567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:42:34.534111 18984 ts_manager.cc:194] Registered new tserver with Master: ba9f35210e774d0faa7f2677fb5be60a (127.0.98.1:37803)
I20260924 17:42:34.535282 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50441
W20260924 17:42:34.616410 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:34.616649 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:34.616675 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:34.617883 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:34.617933 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:34.617960 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:34.619423 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:34.619655 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:34.619936 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:34.625236 19181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.625273 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.625437 19176 server_base.cc:1034] running on GCE node
W20260924 17:42:34.625564 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.625800 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:34.626080 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:34.627233 19176 hybrid_clock.cc:648] HybridClock initialized: now 1790271754627205 us; error 47 us; skew 500 ppm
I20260924 17:42:34.628784 19176 webserver.cc:460] Webserver started at http://127.0.98.2:35311/ using document root <none> and password file <none>
I20260924 17:42:34.629068 19176 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:34.629119 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:34.629254 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:34.630615 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "b2d5ab527213426bbeb406ccb868b490"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "bc68c700551a6c9cfc7aaf31fa8f9616"
server_key_iv: "e689f7faea5e5cb1a59635e77638234e"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.631345 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b2d5ab527213426bbeb406ccb868b490"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "bc68c700551a6c9cfc7aaf31fa8f9616"
server_key_iv: "e689f7faea5e5cb1a59635e77638234e"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.632807 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:34.633713 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.633926 19176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:34.634042 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "b2d5ab527213426bbeb406ccb868b490"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "bc68c700551a6c9cfc7aaf31fa8f9616"
server_key_iv: "e689f7faea5e5cb1a59635e77638234e"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.634126 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:34.643118 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:34.643458 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:34.643613 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:34.643852 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:34.644209 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:34.644264 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.644297 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:34.644325 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.649959 19176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34181
I20260924 17:42:34.650000 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34181 every 8 connection(s)
I20260924 17:42:34.650338 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:42:34.655421 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.651100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44483 (local address 127.0.98.62:34033)
0924 17:42:34.651174 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.651176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.651405 (+ 229us) server_negotiation.cc:407] Connection header received
0924 17:42:34.651879 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.651880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.651890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.651958 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.652610 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.653103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.654109 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.654315 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.654373 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.655193 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.655194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.655195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:34.655199 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.655213 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:34.655216 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.655263 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.655334 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.655356 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:42:34.655646 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.651093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34033 (local address 127.0.98.2:44483)
0924 17:42:34.651266 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.651308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.651786 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.652014 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.652017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.652025 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:34.652524 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.652528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.653188 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.653189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.654020 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.654022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.654394 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.654395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.654503 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.655012 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:34.655013 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:34.655083 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:34.655106 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:34.655266 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:34.655269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:34.655271 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.655322 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.655402 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260924 17:42:34.656179 19304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34033
I20260924 17:42:34.656258 19304 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:34.656472 19304 heartbeater.cc:511] Master 127.0.98.62:34033 requested a full tablet report, sending...
I20260924 17:42:34.656877 18984 ts_manager.cc:194] Registered new tserver with Master: b2d5ab527213426bbeb406ccb868b490 (127.0.98.2:34181)
I20260924 17:42:34.657256 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44483
I20260924 17:42:34.658983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19176
I20260924 17:42:34.659085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:34.659250 392 external_mini_cluster.cc:1428] Setting key 9642ed2a7f3046b6d650851bd0a5bc3c
I20260924 17:42:34.660315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:34.744840 19307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:34.745088 19307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:34.745126 19307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:34.746338 19307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:34.746411 19307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:34.746455 19307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:34.747970 19307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:34.748250 19307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:34.748579 19307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:34.754005 19313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.753995 19315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:34.753995 19312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:34.754034 19307 server_base.cc:1034] running on GCE node
I20260924 17:42:34.754379 19307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:34.754606 19307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:34.755805 19307 hybrid_clock.cc:648] HybridClock initialized: now 1790271754755795 us; error 34 us; skew 500 ppm
I20260924 17:42:34.760689 19307 webserver.cc:460] Webserver started at http://127.0.98.3:39441/ using document root <none> and password file <none>
I20260924 17:42:34.760988 19307 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:34.761067 19307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:34.761228 19307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:34.762423 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "37c92df65f28461797b09dabcf8e4cfa"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "9386de3582c729e297d543248c7f276b"
server_key_iv: "1a4cccfbe44e114d735fb81d541ecd83"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.762817 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37c92df65f28461797b09dabcf8e4cfa"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "9386de3582c729e297d543248c7f276b"
server_key_iv: "1a4cccfbe44e114d735fb81d541ecd83"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.764381 19307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:34.765372 19321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.765652 19307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:34.765774 19307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "37c92df65f28461797b09dabcf8e4cfa"
format_stamp: "Formatted at 2026-09-24 17:42:34 on dist-test-slave-d0pf"
server_key: "9386de3582c729e297d543248c7f276b"
server_key_iv: "1a4cccfbe44e114d735fb81d541ecd83"
server_key_version: "encryptionkey@0"
I20260924 17:42:34.765867 19307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:34.781392 19307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:34.781757 19307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:34.781924 19307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:34.782187 19307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:34.782526 19307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:34.782579 19307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.782620 19307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:34.782652 19307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:34.788877 19307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43047
I20260924 17:42:34.788945 19434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43047 every 8 connection(s)
I20260924 17:42:34.789314 19307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:42:34.794621 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.790157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36259 (local address 127.0.98.62:34033)
0924 17:42:34.790230 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.790233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.790521 (+ 288us) server_negotiation.cc:407] Connection header received
0924 17:42:34.791010 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.791011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.791021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.791093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.791757 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.792249 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.793285 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.793478 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.793549 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.794377 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.794378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.794379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:34.794386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.794401 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:34.794405 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.794454 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.794534 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.794558 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:42:34.794831 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.790138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34033 (local address 127.0.98.3:36259)
0924 17:42:34.790398 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.790435 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.790925 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.791168 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.791171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.791180 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:34.791676 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.791681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.792361 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.792362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.793175 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.793177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.793565 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.793566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.793694 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.794232 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:34.794232 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:34.794262 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:34.794286 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:34.794464 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:34.794468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:34.794471 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.794524 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.794609 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":119,"threads_started":1}
I20260924 17:42:34.795441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19307
I20260924 17:42:34.795517 19435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34033
I20260924 17:42:34.795562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:34.795624 19435 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:34.795807 392 external_mini_cluster.cc:1428] Setting key b9acf41fa8ed03c8bdff690ea6550d41
I20260924 17:42:34.795887 19435 heartbeater.cc:511] Master 127.0.98.62:34033 requested a full tablet report, sending...
I20260924 17:42:34.796319 18984 ts_manager.cc:194] Registered new tserver with Master: 37c92df65f28461797b09dabcf8e4cfa (127.0.98.3:43047)
I20260924 17:42:34.796700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:34.796813 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36259
I20260924 17:42:34.801146 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.798303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.98.62:34033)
0924 17:42:34.798343 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.798345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.798446 (+ 101us) server_negotiation.cc:407] Connection header received
0924 17:42:34.798491 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.798491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.798500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.798540 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:34.798707 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.799209 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.800005 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.800186 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.800268 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.800924 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:34.800925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:34.800925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:34.800929 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:34.800941 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:34.800945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:34.801018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.801080 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.801090 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:42:34.802826 18984 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:34.804409 18984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:34.810887 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.806688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.0.98.3:43047)
0924 17:42:34.807072 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.807076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.807087 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:42:34.807127 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.807129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.807149 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.807314 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.807485 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808649 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.809891 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810478 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810656 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.810760 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.810798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":73,"threads_started":1}
I20260924 17:42:34.810877 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.806597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43047 (local address 127.0.0.1:46332)
0924 17:42:34.806805 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.806850 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.806918 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.807269 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.807272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.807274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.807372 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.807375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808733 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.808734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.809811 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.809813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810560 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.810647 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810660 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.810738 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.810795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":15,"thread_start_us":96,"threads_started":1}
I20260924 17:42:34.811025 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.806822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37803 (local address 127.0.0.1:42824)
0924 17:42:34.807122 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.807131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.807177 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.807554 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.807555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.807557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.807586 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.807588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808868 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.808869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.809855 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.809858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810819 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.810873 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810877 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.810921 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.810954 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260924 17:42:34.811071 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.806698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.2:34181)
0924 17:42:34.806914 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.806919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.806931 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:42:34.807005 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.807008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.807024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.807131 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.807544 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808289 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.809772 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810511 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810583 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810669 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.810929 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.810968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260924 17:42:34.811196 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.806599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34181 (local address 127.0.0.1:48996)
0924 17:42:34.806838 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.806850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.806918 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.807272 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.807273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.807276 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.807467 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.807468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808377 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.808378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.809683 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.809685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810707 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.810783 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810787 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.810838 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.810882 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260924 17:42:34.811275 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.807061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.0.98.1:37803)
0924 17:42:34.807342 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.807346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.807357 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:42:34.807396 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.807399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.807415 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.807487 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.807674 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.808778 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.809949 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.810654 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.810713 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.810788 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.811159 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.811201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260924 17:42:34.811965 19238 tablet_service.cc:1467] Processing CreateTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 (DEFAULT_TABLE table=test-table [id=9aa0e351e30a4eada527847130f494d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.811980 19107 tablet_service.cc:1467] Processing CreateTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 (DEFAULT_TABLE table=test-table [id=9aa0e351e30a4eada527847130f494d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.812331 19107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d2fe7de180470f8eb24263f3ea1b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.812331 19238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d2fe7de180470f8eb24263f3ea1b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.812453 19369 tablet_service.cc:1467] Processing CreateTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 (DEFAULT_TABLE table=test-table [id=9aa0e351e30a4eada527847130f494d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:34.812747 19369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d2fe7de180470f8eb24263f3ea1b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:34.815171 19449 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: Bootstrap starting.
I20260924 17:42:34.815699 19450 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Bootstrap starting.
I20260924 17:42:34.815950 19449 tablet_bootstrap.cc:654] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.816046 19451 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Bootstrap starting.
I20260924 17:42:34.816358 19450 tablet_bootstrap.cc:654] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.816457 19449 log.cc:824] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.816673 19450 log.cc:824] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.816669 19451 tablet_bootstrap.cc:654] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:34.816993 19451 log.cc:824] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:34.817399 19449 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: No bootstrap required, opened a new log
I20260924 17:42:34.817413 19450 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: No bootstrap required, opened a new log
I20260924 17:42:34.817487 19449 ts_tablet_manager.cc:1397] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:34.817507 19450 ts_tablet_manager.cc:1397] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:34.817834 19451 tablet_bootstrap.cc:492] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: No bootstrap required, opened a new log
I20260924 17:42:34.817902 19451 ts_tablet_manager.cc:1397] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:34.819591 19450 raft_consensus.cc:348] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.819751 19450 raft_consensus.cc:374] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.819792 19450 raft_consensus.cc:729] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d5ab527213426bbeb406ccb868b490, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.819924 19450 consensus_queue.cc:260] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.820235 19304 heartbeater.cc:503] Master 127.0.98.62:34033 was elected leader, sending a full tablet report...
I20260924 17:42:34.820181 19449 raft_consensus.cc:348] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.820348 19449 raft_consensus.cc:374] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.820389 19449 raft_consensus.cc:729] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f35210e774d0faa7f2677fb5be60a, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.820530 19450 ts_tablet_manager.cc:1428] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:34.820523 19449 consensus_queue.cc:260] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.820770 19449 ts_tablet_manager.cc:1428] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 17:42:34.820813 19173 heartbeater.cc:503] Master 127.0.98.62:34033 was elected leader, sending a full tablet report...
I20260924 17:42:34.821004 19451 raft_consensus.cc:348] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.821180 19451 raft_consensus.cc:374] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:34.821434 19451 raft_consensus.cc:729] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c92df65f28461797b09dabcf8e4cfa, State: Initialized, Role: FOLLOWER
I20260924 17:42:34.821591 19451 consensus_queue.cc:260] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.821846 19451 ts_tablet_manager.cc:1428] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 17:42:34.821973 19435 heartbeater.cc:503] Master 127.0.98.62:34033 was elected leader, sending a full tablet report...
W20260924 17:42:34.900959 19305 tablet.cc:2369] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:42:34.951264 19455 raft_consensus.cc:482] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:34.951431 19455 raft_consensus.cc:504] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.951890 19455 leader_election.cc:288] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37c92df65f28461797b09dabcf8e4cfa (127.0.98.3:43047), ba9f35210e774d0faa7f2677fb5be60a (127.0.98.1:37803)
I20260924 17:42:34.955852 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.952170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51613 (local address 127.0.98.1:37803)
0924 17:42:34.952235 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.952237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.952307 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:42:34.952376 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.952377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.952387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.952445 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.952588 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.953305 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.954851 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.955443 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.955519 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.955529 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.955735 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.955782 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:42:34.955852 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.952166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55461 (local address 127.0.98.3:43047)
0924 17:42:34.952219 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.952221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.952230 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:34.952259 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.952260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.952268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.952326 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:34.952534 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.953302 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.954850 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.955443 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.955519 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.955529 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.955735 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.955782 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:42:34.955847 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.952035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43047 (local address 127.0.98.2:55461)
0924 17:42:34.952091 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.952102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.952147 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.952345 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.952346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.952348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.952449 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.952451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.953398 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.953399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.954746 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.954750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.955533 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.955534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.955620 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.955626 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.955690 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.955734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":80}
I20260924 17:42:34.955853 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.952064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37803 (local address 127.0.98.2:51613)
0924 17:42:34.952232 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:34.952241 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:34.952319 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:34.952455 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:34.952455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:34.952457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:34.952521 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.952522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.953398 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.953399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.954750 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:34.954752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.955534 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.955535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:34.955620 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.955626 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:42:34.955701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 17:42:34.955734 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":43,"thread_start_us":100,"threads_started":1}
I20260924 17:42:34.956267 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d2fe7de180470f8eb24263f3ea1b39" candidate_uuid: "b2d5ab527213426bbeb406ccb868b490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37c92df65f28461797b09dabcf8e4cfa" is_pre_election: true
I20260924 17:42:34.956450 19389 raft_consensus.cc:2454] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d5ab527213426bbeb406ccb868b490 in term 0.
I20260924 17:42:34.956507 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d2fe7de180470f8eb24263f3ea1b39" candidate_uuid: "b2d5ab527213426bbeb406ccb868b490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f35210e774d0faa7f2677fb5be60a" is_pre_election: true
I20260924 17:42:34.956667 19127 raft_consensus.cc:2454] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d5ab527213426bbeb406ccb868b490 in term 0.
I20260924 17:42:34.956732 19194 leader_election.cc:302] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37c92df65f28461797b09dabcf8e4cfa, b2d5ab527213426bbeb406ccb868b490; no voters:
I20260924 17:42:34.956846 19455 raft_consensus.cc:2781] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:34.956897 19455 raft_consensus.cc:482] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:34.956931 19455 raft_consensus.cc:3037] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.957650 19455 raft_consensus.cc:504] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.957814 19455 leader_election.cc:288] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [CANDIDATE]: Term 1 election: Requested vote from peers 37c92df65f28461797b09dabcf8e4cfa (127.0.98.3:43047), ba9f35210e774d0faa7f2677fb5be60a (127.0.98.1:37803)
I20260924 17:42:34.957975 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d2fe7de180470f8eb24263f3ea1b39" candidate_uuid: "b2d5ab527213426bbeb406ccb868b490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f35210e774d0faa7f2677fb5be60a"
I20260924 17:42:34.958003 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d2fe7de180470f8eb24263f3ea1b39" candidate_uuid: "b2d5ab527213426bbeb406ccb868b490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37c92df65f28461797b09dabcf8e4cfa"
I20260924 17:42:34.958076 19127 raft_consensus.cc:3037] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.958104 19389 raft_consensus.cc:3037] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:34.958845 19389 raft_consensus.cc:2454] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d5ab527213426bbeb406ccb868b490 in term 1.
I20260924 17:42:34.958851 19127 raft_consensus.cc:2454] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d5ab527213426bbeb406ccb868b490 in term 1.
I20260924 17:42:34.959012 19194 leader_election.cc:302] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37c92df65f28461797b09dabcf8e4cfa, b2d5ab527213426bbeb406ccb868b490; no voters:
I20260924 17:42:34.959159 19455 raft_consensus.cc:2781] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:34.959321 19455 raft_consensus.cc:686] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 1 LEADER]: Becoming Leader. State: Replica: b2d5ab527213426bbeb406ccb868b490, State: Running, Role: LEADER
I20260924 17:42:34.959415 19455 consensus_queue.cc:237] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } }
I20260924 17:42:34.960279 18985 catalog_manager.cc:5495] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2d5ab527213426bbeb406ccb868b490 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b2d5ab527213426bbeb406ccb868b490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d5ab527213426bbeb406ccb868b490" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 } health_report { overall_health: UNKNOWN } } }
I20260924 17:42:34.993268 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:34.990472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.0.98.2:34181)
0924 17:42:34.990554 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:34.990556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:34.990572 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:42:34.990609 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:34.990610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:34.990630 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:34.990694 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:34.990839 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.991399 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.992353 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:34.992553 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:34.992617 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:34.992813 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:34.992904 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:34.992982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:34.993058 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 17:42:34.993095 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 17:42:34.995795 19389 raft_consensus.cc:1261] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 1 FOLLOWER]: Refusing update from remote peer b2d5ab527213426bbeb406ccb868b490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:34.996263 19127 raft_consensus.cc:1261] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 1 FOLLOWER]: Refusing update from remote peer b2d5ab527213426bbeb406ccb868b490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:34.996312 19455 consensus_queue.cc:1035] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c92df65f28461797b09dabcf8e4cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:34.996582 19455 consensus_queue.cc:1035] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9f35210e774d0faa7f2677fb5be60a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:35.000679 19463 mvcc.cc:205] Tried to move back new op lower bound from 7332953108460298240 to 7332953108314763264. Current Snapshot: MvccSnapshot[applied={T|T < 7332953108460298240}]
I20260924 17:42:35.002148 18985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:35.002270 18985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:35.003082 18985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 582488a2974d4bb1ae07cac66fc90d54: Sending DeleteTablet for 3 replicas of tablet 77d2fe7de180470f8eb24263f3ea1b39
I20260924 17:42:35.003432 19238 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='slave'} at 127.0.0.1:48996
I20260924 17:42:35.003520 19107 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='slave'} at 127.0.0.1:42824
I20260924 17:42:35.003566 19369 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d2fe7de180470f8eb24263f3ea1b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:34 UTC) from {username='slave'} at 127.0.0.1:46332
I20260924 17:42:35.003796 19472 tablet_replica.cc:330] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a: stopping tablet replica
I20260924 17:42:35.003909 19472 raft_consensus.cc:2229] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:35.004009 19471 tablet_replica.cc:330] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: stopping tablet replica
I20260924 17:42:35.004040 19472 raft_consensus.cc:2258] T 77d2fe7de180470f8eb24263f3ea1b39 P ba9f35210e774d0faa7f2677fb5be60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:35.004122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19045
I20260924 17:42:35.004216 19473 tablet_replica.cc:330] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: stopping tablet replica
I20260924 17:42:35.004885 19473 raft_consensus.cc:2229] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:35.004989 19473 raft_consensus.cc:2258] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:35.005451 19473 ts_tablet_manager.cc:1905] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:35.005797 19471 raft_consensus.cc:2229] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:42:35.005954 19471 raft_consensus.cc:2258] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:35.006449 19471 ts_tablet_manager.cc:1905] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:35.007369 19473 ts_tablet_manager.cc:1918] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:35.007449 19473 log.cc:1196] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/77d2fe7de180470f8eb24263f3ea1b39
I20260924 17:42:35.007722 19473 ts_tablet_manager.cc:1939] T 77d2fe7de180470f8eb24263f3ea1b39 P 37c92df65f28461797b09dabcf8e4cfa: Deleting consensus metadata
I20260924 17:42:35.008774 19471 ts_tablet_manager.cc:1918] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:35.008859 19471 log.cc:1196] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/77d2fe7de180470f8eb24263f3ea1b39
I20260924 17:42:35.009094 19471 ts_tablet_manager.cc:1939] T 77d2fe7de180470f8eb24263f3ea1b39 P b2d5ab527213426bbeb406ccb868b490: Deleting consensus metadata
I20260924 17:42:35.009600 18970 catalog_manager.cc:4841] TS b2d5ab527213426bbeb406ccb868b490 (127.0.98.2:34181): tablet 77d2fe7de180470f8eb24263f3ea1b39 (table test-table [id=9aa0e351e30a4eada527847130f494d4]) successfully deleted
W20260924 17:42:35.010051 18970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37803 (error 108)
I20260924 17:42:35.010648 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:35.010514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37803 (local address 127.0.0.1:42838)
0924 17:42:35.010563 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:35.010602 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 17:42:35.010856 18970 catalog_manager.cc:4584] TS ba9f35210e774d0faa7f2677fb5be60a (127.0.98.1:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77d2fe7de180470f8eb24263f3ea1b39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37803: connect: Connection refused (error 111)
I20260924 17:42:35.011323 18972 catalog_manager.cc:4841] TS 37c92df65f28461797b09dabcf8e4cfa (127.0.98.3:43047): tablet 77d2fe7de180470f8eb24263f3ea1b39 (table test-table [id=9aa0e351e30a4eada527847130f494d4]) successfully deleted
I20260924 17:42:35.011783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19176
I20260924 17:42:35.016803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19307
I20260924 17:42:35.021600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 18954
2026-09-24T17:42:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (791 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:35 dist-test-slave-d0pf krb5kdc[19477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:35 dist-test-slave-d0pf krb5kdc[19477](info): set up 2 sockets
Sep 24 17:42:35 dist-test-slave-d0pf krb5kdc[19477](info): commencing operation
krb5kdc: starting...
W20260924 17:42:36.062844 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:36.180076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:36.261667 19490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:36.261940 19490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:36.261979 19490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:36.263226 19490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:36.263294 19490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:36.263326 19490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:36.263358 19490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:36.263384 19490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:36.264904 19490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:36.265188 19490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:36.265426 19490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:36.270349 19496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.270395 19495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.270501 19490 server_base.cc:1034] running on GCE node
W20260924 17:42:36.270589 19498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.270869 19490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:36.271251 19490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:36.272480 19490 hybrid_clock.cc:648] HybridClock initialized: now 1790271756272459 us; error 34 us; skew 500 ppm
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:36.275156 19490 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:36.275629 19490 webserver.cc:460] Webserver started at http://127.0.98.62:38803/ using document root <none> and password file <none>
I20260924 17:42:36.275867 19490 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:36.275913 19490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:36.276059 19490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:36.277132 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "2cf105dc9585487885d6bd6e2097a56d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4bc224292c5aace5d84e03c3dfd55209"
server_key_iv: "4ccaa75b97d08615425c9523a47e1074"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.277511 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cf105dc9585487885d6bd6e2097a56d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4bc224292c5aace5d84e03c3dfd55209"
server_key_iv: "4ccaa75b97d08615425c9523a47e1074"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.278867 19490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:36.279693 19505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.279906 19490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:36.280014 19490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "2cf105dc9585487885d6bd6e2097a56d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4bc224292c5aace5d84e03c3dfd55209"
server_key_iv: "4ccaa75b97d08615425c9523a47e1074"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.280092 19490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:36.313804 19490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:36.315826 19490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:36.315975 19490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:36.319878 19490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36299
I20260924 17:42:36.319912 19557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36299 every 8 connection(s)
I20260924 17:42:36.320281 19490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:36.321126 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:36.323559 19558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: Bootstrap starting.
I20260924 17:42:36.324147 19558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:36.324434 19558 log.cc:824] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:36.325101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19490
I20260924 17:42:36.325150 19558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: No bootstrap required, opened a new log
I20260924 17:42:36.325234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:36.325426 392 external_mini_cluster.cc:1428] Setting key 61e80e03067086cff26429e9f5ff7823
I20260924 17:42:36.327135 19558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260924 17:42:36.327299 19558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:36.327360 19558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf105dc9585487885d6bd6e2097a56d, State: Initialized, Role: FOLLOWER
I20260924 17:42:36.327533 19558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260924 17:42:36.327626 19558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:36.327663 19558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:36.327704 19558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:36.328719 19558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260924 17:42:36.328850 19558 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf105dc9585487885d6bd6e2097a56d; no voters:
I20260924 17:42:36.329037 19558 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:36.329063 19563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:36.329167 19563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [term 1 LEADER]: Becoming Leader. State: Replica: 2cf105dc9585487885d6bd6e2097a56d, State: Running, Role: LEADER
I20260924 17:42:36.329299 19558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:36.329320 19563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260924 17:42:36.329665 19563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf105dc9585487885d6bd6e2097a56d. Latest consensus state: current_term: 1 leader_uuid: "2cf105dc9585487885d6bd6e2097a56d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } }
I20260924 17:42:36.329747 19563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:36.330051 19568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:36.330657 19568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:36.330953 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf105dc9585487885d6bd6e2097a56d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf105dc9585487885d6bd6e2097a56d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } }
I20260924 17:42:36.331334 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:36.332731 19568 catalog_manager.cc:1347] Generated new cluster ID: 82f93a325b0c45d88225b9b43307d7cf
I20260924 17:42:36.332813 19568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:36.337853 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.325937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.0.98.62:36299)
0924 17:42:36.326131 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.326138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.326163 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:42:36.327117 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.327134 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.327189 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.327311 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:36.328189 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.329273 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.330468 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.331115 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.331548 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.334127 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:36.334131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:36.334131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:36.334143 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:36.336159 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.336571 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.336573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.336574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.336692 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.336913 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.336914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.336914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.337219 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:36.337369 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.337526 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.337659 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260924 17:42:36.342620 19568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:36.343348 19568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:36.346614 19568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: Generated new TSK 0
I20260924 17:42:36.346791 19568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 17:42:36.373898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:36.456521 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:36.456740 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:36.456765 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:36.457888 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:36.457929 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:36.457957 19584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:36.459561 19584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:36.459779 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:36.459971 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:36.465265 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.465272 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.465305 19584 server_base.cc:1034] running on GCE node
W20260924 17:42:36.465278 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.465659 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:36.466049 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:36.467224 19584 hybrid_clock.cc:648] HybridClock initialized: now 1790271756467212 us; error 46 us; skew 500 ppm
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:36.469907 19584 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 17:42:36.470357 19584 webserver.cc:460] Webserver started at http://127.0.98.1:37889/ using document root <none> and password file <none>
I20260924 17:42:36.470595 19584 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:36.470665 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:36.470811 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:36.471936 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "360eb4801b2648f9b2679f90b581b3a2"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4e6c7960d7917f9a44cf4b6d0df887dd"
server_key_iv: "6fbdec1e95ae6b8325b6c48a4f8fda3e"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.472301 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "360eb4801b2648f9b2679f90b581b3a2"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4e6c7960d7917f9a44cf4b6d0df887dd"
server_key_iv: "6fbdec1e95ae6b8325b6c48a4f8fda3e"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.473672 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:42:36.474483 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.474702 19584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:36.474805 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "360eb4801b2648f9b2679f90b581b3a2"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "4e6c7960d7917f9a44cf4b6d0df887dd"
server_key_iv: "6fbdec1e95ae6b8325b6c48a4f8fda3e"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.474889 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:36.485761 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:36.487695 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:36.487857 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:36.488086 19584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:36.488421 19584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:36.488477 19584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.488517 19584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:36.488559 19584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.494379 19584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42799
I20260924 17:42:36.494421 19712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42799 every 8 connection(s)
I20260924 17:42:36.494769 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:36.498893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19584
I20260924 17:42:36.499022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:36.499276 392 external_mini_cluster.cc:1428] Setting key 6446534afdbb55b06ee5614727d2adf7
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:36.505053 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.495576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34413 (local address 127.0.98.62:36299)
0924 17:42:36.495624 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.495626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.495818 (+ 192us) server_negotiation.cc:407] Connection header received
0924 17:42:36.496798 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.496799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.496816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.496873 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:36.497500 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.498012 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.499074 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.499307 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.499397 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.501842 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:36.501844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:36.501845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:36.501857 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:36.503589 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.504019 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.504020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.504021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.504080 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.504347 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.504348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.504349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.504473 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:36.504542 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.504834 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.504959 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:42:36.505224 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.495532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.1:34413)
0924 17:42:36.495685 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.495727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.496655 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.496942 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.496946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.497206 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:36.497397 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.497401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.498116 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.498117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.498962 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.498964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.499408 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.499409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.499543 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.500197 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:36.500208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:36.501745 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:36.503686 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.503694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.503696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.503925 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.504154 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.504155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.504156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.504270 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.504545 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:36.504553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:36.504701 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.504812 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.504977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 17:42:36.505712 19713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260924 17:42:36.505828 19713 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:36.506042 19713 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260924 17:42:36.506583 19522 ts_manager.cc:194] Registered new tserver with Master: 360eb4801b2648f9b2679f90b581b3a2 (127.0.98.1:42799)
I20260924 17:42:36.507336 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34413
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 17:42:36.526264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:36.612408 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:36.612613 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:36.612640 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:36.613852 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:36.613895 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:36.613925 19718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:36.615475 19718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:36.615690 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:36.615916 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:36.620893 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.620895 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.620906 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.621043 19718 server_base.cc:1034] running on GCE node
I20260924 17:42:36.621359 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:36.621611 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:36.622758 19718 hybrid_clock.cc:648] HybridClock initialized: now 1790271756622747 us; error 27 us; skew 500 ppm
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:36.625440 19718 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 17:42:36.625912 19718 webserver.cc:460] Webserver started at http://127.0.98.2:33939/ using document root <none> and password file <none>
I20260924 17:42:36.626116 19718 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:36.626156 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:36.626245 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:36.627364 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad53584b3c614b26aac77963855fd96d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "534fbdc1b38962fbdb8b8dad534d7075"
server_key_iv: "3dd5b840e36d16c1636ced823808e35f"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.627673 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad53584b3c614b26aac77963855fd96d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "534fbdc1b38962fbdb8b8dad534d7075"
server_key_iv: "3dd5b840e36d16c1636ced823808e35f"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.629006 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:36.629908 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.630110 19718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:36.630215 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "ad53584b3c614b26aac77963855fd96d"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "534fbdc1b38962fbdb8b8dad534d7075"
server_key_iv: "3dd5b840e36d16c1636ced823808e35f"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.630348 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:36.648561 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:36.650538 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:36.650710 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:36.650947 19718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:36.651314 19718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:36.651372 19718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.651407 19718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:36.651435 19718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.657241 19718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45287
I20260924 17:42:36.657294 19846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45287 every 8 connection(s)
I20260924 17:42:36.657640 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:42:36.661724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19718
I20260924 17:42:36.661824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:36.662036 392 external_mini_cluster.cc:1428] Setting key 796597eb99a348d1f1a1a78779675a5f
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:36.668710 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.658380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.2:55961)
0924 17:42:36.658579 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.658616 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.659684 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.659968 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.659972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.660247 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:36.660450 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.660454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.661170 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.661171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.662407 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.662410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.662825 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.662826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.662995 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.663561 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:36.663571 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:36.665450 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:36.667337 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.667343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.667345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.667560 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.667789 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.667790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.667790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.667893 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.668207 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:36.668211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:36.668330 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.668411 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.668538 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260924 17:42:36.668709 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.658414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55961 (local address 127.0.98.62:36299)
0924 17:42:36.658460 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.658462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.658704 (+ 242us) server_negotiation.cc:407] Connection header received
0924 17:42:36.659823 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.659825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.659840 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.659888 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:36.660558 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.661070 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.662497 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.662719 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.662798 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.665555 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:36.665556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:36.665557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:36.665565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:36.667257 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.667643 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.667645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.667645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.667716 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.667968 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.667969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.667969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.668135 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:36.668225 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.668469 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.668568 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:42:36.669260 19847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260924 17:42:36.669373 19847 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:36.669582 19847 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260924 17:42:36.670037 19522 ts_manager.cc:194] Registered new tserver with Master: ad53584b3c614b26aac77963855fd96d (127.0.98.2:45287)
I20260924 17:42:36.670439 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 17:42:36.689558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:36.775940 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:36.776156 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:36.776181 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:36.777311 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:36.777348 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:36.777374 19852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:36.778927 19852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:36.779197 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:36.779400 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:36.784602 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.784626 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:36.784610 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:36.784854 19852 server_base.cc:1034] running on GCE node
I20260924 17:42:36.785063 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:36.785334 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:36.786501 19852 hybrid_clock.cc:648] HybridClock initialized: now 1790271756786500 us; error 42 us; skew 500 ppm
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:36.789114 19852 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 17:42:36.789608 19852 webserver.cc:460] Webserver started at http://127.0.98.3:44951/ using document root <none> and password file <none>
I20260924 17:42:36.789795 19852 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:36.789835 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:36.789927 19852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:36.790939 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b3feb579cca430289a20213e43d1cab"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "ea7390133cf7c43fb7de00550ed2ce4d"
server_key_iv: "be0152cce68a8334aa29d2b9bf3147a7"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.791325 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b3feb579cca430289a20213e43d1cab"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "ea7390133cf7c43fb7de00550ed2ce4d"
server_key_iv: "be0152cce68a8334aa29d2b9bf3147a7"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.792616 19852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:36.793334 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.793526 19852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:36.793635 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "3b3feb579cca430289a20213e43d1cab"
format_stamp: "Formatted at 2026-09-24 17:42:36 on dist-test-slave-d0pf"
server_key: "ea7390133cf7c43fb7de00550ed2ce4d"
server_key_iv: "be0152cce68a8334aa29d2b9bf3147a7"
server_key_version: "encryptionkey@0"
I20260924 17:42:36.793694 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:36.806001 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:36.808394 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:36.808514 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:36.808709 19852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:36.809038 19852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:36.809073 19852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.809128 19852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:36.809155 19852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:36.814869 19852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44681
I20260924 17:42:36.814963 19980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44681 every 8 connection(s)
I20260924 17:42:36.815315 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:42:36 dist-test-slave-d0pf krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:36.824513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 19852
I20260924 17:42:36.824564 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.816135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46247 (local address 127.0.98.62:36299)
0924 17:42:36.816184 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.816186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.816381 (+ 195us) server_negotiation.cc:407] Connection header received
0924 17:42:36.817340 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.817342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.817357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.817441 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:36.818098 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.818624 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.819628 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.819815 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.819878 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.822081 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:36.822083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:36.822083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:36.822089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:36.823369 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.823745 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.823746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.823746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.823795 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.824003 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.824004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.824004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.824107 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:36.824156 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.824439 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.824499 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:42:36.824632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:36.824831 392 external_mini_cluster.cc:1428] Setting key c059ba3916ddee159df42a7f24f8e467
I20260924 17:42:36.824863 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.816075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.3:46247)
0924 17:42:36.816248 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.816290 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.817214 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.817505 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.817509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.817771 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:36.817987 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.817991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.818711 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.818712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.819526 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.819528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.819896 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.819896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.820007 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.820583 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:42:36.820592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:36.821995 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:36.823448 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.823453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.823455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.823682 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.823852 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:42:36.823853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:42:36.823853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:42:36.823950 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:42:36.824162 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:36.824166 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:36.824282 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.824421 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.824624 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260924 17:42:36.825392 19981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260924 17:42:36.825532 19981 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:36.825726 19981 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260924 17:42:36.826112 19522 ts_manager.cc:194] Registered new tserver with Master: 3b3feb579cca430289a20213e43d1cab (127.0.98.3:44681)
I20260924 17:42:36.826494 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46247
I20260924 17:42:36.836898 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:36.844523 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.838389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.98.62:36299)
0924 17:42:36.838444 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.838446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.838499 (+ 53us) server_negotiation.cc:407] Connection header received
0924 17:42:36.838551 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.838551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.838560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.838615 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:36.839231 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.839747 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.840426 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.840609 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.840658 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.841812 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:36.841813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:36.841814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:36.841820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:36.843542 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.843871 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.843871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.843872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.843923 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:36.844083 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:36.844084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:36.844084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:36.844228 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:36.844299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.844401 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.844457 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:42:36.846113 19522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:36.847770 19522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:36.854135 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.849984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44681 (local address 127.0.0.1:33946)
0924 17:42:36.850234 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.850279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.850350 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.850674 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.850677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.850679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:36.850748 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.850750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.852108 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.852109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.853169 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.853171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.853818 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.853896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.853911 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.853978 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.854037 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260924 17:42:36.854293 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.849994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.0.1:50218)
0924 17:42:36.850450 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.850459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.850494 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.850879 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.850880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.850881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:36.850930 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.850932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.852149 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.852151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.853048 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.853051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.854011 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.854012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.854097 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.854101 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.854154 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.854191 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":79,"threads_started":1}
I20260924 17:42:36.854377 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.849985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45287 (local address 127.0.0.1:34392)
0924 17:42:36.850245 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.850285 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.850349 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.850833 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.850834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.850837 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:36.850898 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.850900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.851851 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.851852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.853126 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.853129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.853981 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.854048 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.854051 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.854108 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.854150 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":56,"threads_started":1}
I20260924 17:42:36.854400 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.850220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.0.98.3:44681)
0924 17:42:36.850399 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.850403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.850414 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:42:36.850455 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.850459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.850478 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.850574 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:36.850838 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.851996 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853255 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.853728 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.853881 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.854270 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.854323 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260924 17:42:36.854602 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.850420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.98.1:42799)
0924 17:42:36.850602 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.850607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.850619 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:42:36.850688 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.850690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.850705 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.850792 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:36.850995 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.852064 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853132 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.853841 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853915 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.853974 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.854453 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.854509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260924 17:42:36.855150 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.850080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.0.98.2:45287)
0924 17:42:36.850657 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.850661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.850670 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:36.850709 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.850710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.850726 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.850978 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:36.851075 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.851733 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853215 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.853909 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.853968 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.854018 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.854124 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.854171 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":83,"threads_started":1}
I20260924 17:42:36.855270 19647 tablet_service.cc:1467] Processing CreateTablet for tablet fc519e58f62e40f68a1ddc395635ee99 (DEFAULT_TABLE table=test-table [id=230a7b1534164d54a51e778e877dc877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:36.855271 19915 tablet_service.cc:1467] Processing CreateTablet for tablet fc519e58f62e40f68a1ddc395635ee99 (DEFAULT_TABLE table=test-table [id=230a7b1534164d54a51e778e877dc877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:36.855546 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc519e58f62e40f68a1ddc395635ee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:36.855552 19915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc519e58f62e40f68a1ddc395635ee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:36.857906 19998 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: Bootstrap starting.
I20260924 17:42:36.858072 19999 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Bootstrap starting.
I20260924 17:42:36.858533 19998 tablet_bootstrap.cc:654] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:36.858724 19999 tablet_bootstrap.cc:654] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:36.858651 19781 tablet_service.cc:1467] Processing CreateTablet for tablet fc519e58f62e40f68a1ddc395635ee99 (DEFAULT_TABLE table=test-table [id=230a7b1534164d54a51e778e877dc877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:36.858947 19998 log.cc:824] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:36.859002 19781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc519e58f62e40f68a1ddc395635ee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:36.859304 19999 log.cc:824] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:36.859978 19998 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: No bootstrap required, opened a new log
I20260924 17:42:36.859987 19999 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: No bootstrap required, opened a new log
I20260924 17:42:36.860054 19998 ts_tablet_manager.cc:1397] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:36.860070 19999 ts_tablet_manager.cc:1397] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:36.861518 19999 raft_consensus.cc:348] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.861747 19999 raft_consensus.cc:374] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:36.861797 19999 raft_consensus.cc:729] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3feb579cca430289a20213e43d1cab, State: Initialized, Role: FOLLOWER
I20260924 17:42:36.861925 19999 consensus_queue.cc:260] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.862212 19999 ts_tablet_manager.cc:1428] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:36.862111 19998 raft_consensus.cc:348] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.862279 19998 raft_consensus.cc:374] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:36.862324 19998 raft_consensus.cc:729] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360eb4801b2648f9b2679f90b581b3a2, State: Initialized, Role: FOLLOWER
I20260924 17:42:36.862345 19981 heartbeater.cc:503] Master 127.0.98.62:36299 was elected leader, sending a full tablet report...
I20260924 17:42:36.862426 19998 consensus_queue.cc:260] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.862686 20003 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Bootstrap starting.
I20260924 17:42:36.862712 19998 ts_tablet_manager.cc:1428] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:36.862871 19713 heartbeater.cc:503] Master 127.0.98.62:36299 was elected leader, sending a full tablet report...
I20260924 17:42:36.863451 20003 tablet_bootstrap.cc:654] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:36.863969 20003 log.cc:824] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:36.864550 20003 tablet_bootstrap.cc:492] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: No bootstrap required, opened a new log
I20260924 17:42:36.864600 20003 ts_tablet_manager.cc:1397] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:36.866119 20003 raft_consensus.cc:348] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.866261 20003 raft_consensus.cc:374] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:36.866283 20003 raft_consensus.cc:729] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad53584b3c614b26aac77963855fd96d, State: Initialized, Role: FOLLOWER
I20260924 17:42:36.866357 20003 consensus_queue.cc:260] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.866597 20003 ts_tablet_manager.cc:1428] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:42:36.866652 19847 heartbeater.cc:503] Master 127.0.98.62:36299 was elected leader, sending a full tablet report...
W20260924 17:42:36.908239 19848 tablet.cc:2369] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:42:36.923332 20002 raft_consensus.cc:482] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:36.923496 20002 raft_consensus.cc:504] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.923902 20002 leader_election.cc:288] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad53584b3c614b26aac77963855fd96d (127.0.98.2:45287), 360eb4801b2648f9b2679f90b581b3a2 (127.0.98.1:42799)
I20260924 17:42:36.926622 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.924128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34095 (local address 127.0.98.1:42799)
0924 17:42:36.924172 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.924174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.924183 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:36.924213 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.924213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.924225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.924304 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:36.924476 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.924953 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.925901 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.926267 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.926350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.926360 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.926522 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.926559 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:42:36.926621 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.924052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.98.3:34095)
0924 17:42:36.924104 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.924115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.924155 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.924308 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.924309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.924311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:36.924392 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.924394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.925056 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.925057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.925802 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.925804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.926349 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.926350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.926432 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.926440 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.926503 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.926547 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:42:36.926892 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.924073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45287 (local address 127.0.98.3:57935)
0924 17:42:36.924248 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.924258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:36.924352 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:36.924511 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:36.924511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:36.924513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:36.924586 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.924588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.925245 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.925246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.926003 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:36.926005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.926570 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.926571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:36.926648 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.926651 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:36.926763 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 17:42:36.926814 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260924 17:42:36.926896 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.924136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57935 (local address 127.0.98.2:45287)
0924 17:42:36.924182 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.924184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.924342 (+ 158us) server_negotiation.cc:407] Connection header received
0924 17:42:36.924420 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.924422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.924435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.924507 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:36.924670 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.925150 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.926101 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.926470 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.926564 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.926573 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.926781 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.926832 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:42:36.926951 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc519e58f62e40f68a1ddc395635ee99" candidate_uuid: "3b3feb579cca430289a20213e43d1cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360eb4801b2648f9b2679f90b581b3a2" is_pre_election: true
I20260924 17:42:36.927138 19667 raft_consensus.cc:2454] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3feb579cca430289a20213e43d1cab in term 0.
I20260924 17:42:36.927253 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc519e58f62e40f68a1ddc395635ee99" candidate_uuid: "3b3feb579cca430289a20213e43d1cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad53584b3c614b26aac77963855fd96d" is_pre_election: true
I20260924 17:42:36.927395 19801 raft_consensus.cc:2454] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3feb579cca430289a20213e43d1cab in term 0.
I20260924 17:42:36.927398 19869 leader_election.cc:302] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360eb4801b2648f9b2679f90b581b3a2, 3b3feb579cca430289a20213e43d1cab; no voters:
I20260924 17:42:36.927556 20002 raft_consensus.cc:2781] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:36.927613 20002 raft_consensus.cc:482] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:36.927634 20002 raft_consensus.cc:3037] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:36.928332 20002 raft_consensus.cc:504] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.928493 20002 leader_election.cc:288] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [CANDIDATE]: Term 1 election: Requested vote from peers ad53584b3c614b26aac77963855fd96d (127.0.98.2:45287), 360eb4801b2648f9b2679f90b581b3a2 (127.0.98.1:42799)
I20260924 17:42:36.928639 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc519e58f62e40f68a1ddc395635ee99" candidate_uuid: "3b3feb579cca430289a20213e43d1cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360eb4801b2648f9b2679f90b581b3a2"
I20260924 17:42:36.928699 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc519e58f62e40f68a1ddc395635ee99" candidate_uuid: "3b3feb579cca430289a20213e43d1cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad53584b3c614b26aac77963855fd96d"
I20260924 17:42:36.928747 19667 raft_consensus.cc:3037] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:36.928761 19801 raft_consensus.cc:3037] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:36.929665 19801 raft_consensus.cc:2454] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3feb579cca430289a20213e43d1cab in term 1.
I20260924 17:42:36.929665 19667 raft_consensus.cc:2454] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3feb579cca430289a20213e43d1cab in term 1.
I20260924 17:42:36.929841 19871 leader_election.cc:302] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3feb579cca430289a20213e43d1cab, ad53584b3c614b26aac77963855fd96d; no voters:
I20260924 17:42:36.929955 20002 raft_consensus.cc:2781] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:36.930112 20002 raft_consensus.cc:686] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 1 LEADER]: Becoming Leader. State: Replica: 3b3feb579cca430289a20213e43d1cab, State: Running, Role: LEADER
I20260924 17:42:36.930214 20002 consensus_queue.cc:237] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } }
I20260924 17:42:36.931190 19522 catalog_manager.cc:5495] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b3feb579cca430289a20213e43d1cab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b3feb579cca430289a20213e43d1cab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3feb579cca430289a20213e43d1cab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } health_report { overall_health: UNKNOWN } } }
I20260924 17:42:36.946610 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.944260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.98.3:44681)
0924 17:42:36.944308 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:36.944310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:36.944319 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:36.944392 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:36.944392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:36.944410 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:36.944472 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:36.944592 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.945106 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.945832 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:36.946034 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:36.946088 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:36.946193 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:36.946292 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:36.946354 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:36.946436 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:42:36.946452 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:42:36.948536 19801 raft_consensus.cc:1261] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 1 FOLLOWER]: Refusing update from remote peer 3b3feb579cca430289a20213e43d1cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:36.948527 19667 raft_consensus.cc:1261] T fc519e58f62e40f68a1ddc395635ee99 P 360eb4801b2648f9b2679f90b581b3a2 [term 1 FOLLOWER]: Refusing update from remote peer 3b3feb579cca430289a20213e43d1cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:36.948904 20002 consensus_queue.cc:1035] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb4801b2648f9b2679f90b581b3a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:36.949062 20008 consensus_queue.cc:1035] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad53584b3c614b26aac77963855fd96d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:36.951835 20013 mvcc.cc:205] Tried to move back new op lower bound from 7332953116458770432 to 7332953116387119104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953116458770432}]
I20260924 17:42:36.951942 20012 mvcc.cc:205] Tried to move back new op lower bound from 7332953116458770432 to 7332953116387119104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953116458770432}]
I20260924 17:42:36.952059 20011 mvcc.cc:205] Tried to move back new op lower bound from 7332953116458770432 to 7332953116387119104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953116458770432 or (T in {7332953116458770432})}]
I20260924 17:42:36.953889 19522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:36.953977 19522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:36.954767 19522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf105dc9585487885d6bd6e2097a56d: Sending DeleteTablet for 3 replicas of tablet fc519e58f62e40f68a1ddc395635ee99
I20260924 17:42:36.955130 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet fc519e58f62e40f68a1ddc395635ee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50218
I20260924 17:42:36.955147 19915 tablet_service.cc:1514] Processing DeleteTablet for tablet fc519e58f62e40f68a1ddc395635ee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33946
I20260924 17:42:36.955214 19781 tablet_service.cc:1514] Processing DeleteTablet for tablet fc519e58f62e40f68a1ddc395635ee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34392
I20260924 17:42:36.955387 20022 tablet_replica.cc:330] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: stopping tablet replica
I20260924 17:42:36.955485 20022 raft_consensus.cc:2229] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:42:36.955607 20023 tablet_replica.cc:330] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: stopping tablet replica
I20260924 17:42:36.955621 20022 raft_consensus.cc:2258] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:36.955737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19584
I20260924 17:42:36.955770 20023 raft_consensus.cc:2229] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:36.956557 20023 raft_consensus.cc:2258] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:36.958078 20023 ts_tablet_manager.cc:1905] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:36.959317 20022 ts_tablet_manager.cc:1905] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:36.959730 20023 ts_tablet_manager.cc:1918] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:36.959810 20023 log.cc:1196] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/fc519e58f62e40f68a1ddc395635ee99
I20260924 17:42:36.960125 20023 ts_tablet_manager.cc:1939] T fc519e58f62e40f68a1ddc395635ee99 P ad53584b3c614b26aac77963855fd96d: Deleting consensus metadata
I20260924 17:42:36.960614 19509 catalog_manager.cc:4841] TS ad53584b3c614b26aac77963855fd96d (127.0.98.2:45287): tablet fc519e58f62e40f68a1ddc395635ee99 (table test-table [id=230a7b1534164d54a51e778e877dc877]) successfully deleted
I20260924 17:42:36.961374 20022 ts_tablet_manager.cc:1918] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:36.961473 20022 log.cc:1196] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/fc519e58f62e40f68a1ddc395635ee99
I20260924 17:42:36.961750 20022 ts_tablet_manager.cc:1939] T fc519e58f62e40f68a1ddc395635ee99 P 3b3feb579cca430289a20213e43d1cab: Deleting consensus metadata
I20260924 17:42:36.962162 19508 catalog_manager.cc:4841] TS 3b3feb579cca430289a20213e43d1cab (127.0.98.3:44681): tablet fc519e58f62e40f68a1ddc395635ee99 (table test-table [id=230a7b1534164d54a51e778e877dc877]) successfully deleted
W20260924 17:42:36.962312 19507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42799 (error 108)
I20260924 17:42:36.962874 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:36.962732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.0.1:50222)
0924 17:42:36.962793 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:36.962833 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 17:42:36.962906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19718
W20260924 17:42:36.965304 19507 catalog_manager.cc:4584] TS 360eb4801b2648f9b2679f90b581b3a2 (127.0.98.1:42799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc519e58f62e40f68a1ddc395635ee99: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42799: connect: Connection refused (error 111)
I20260924 17:42:36.968917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19852
I20260924 17:42:36.973136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 19490
2026-09-24T17:42:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1952 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T17:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:36Z Disabled control of system clock
I20260924 17:42:37.001745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39409
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:37.086025 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:37.086261 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:37.086282 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:37.087646 20028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:37.087692 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:37.087705 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:37.087718 20028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:37.087730 20028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:37.089046 20028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:37.089246 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:37.089481 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:37.094203 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.094264 20033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.094286 20034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:37.094393 20028 server_base.cc:1034] running on GCE node
I20260924 17:42:37.094707 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:37.094991 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:37.096168 20028 hybrid_clock.cc:648] HybridClock initialized: now 1790271757096158 us; error 27 us; skew 500 ppm
I20260924 17:42:37.097440 20028 webserver.cc:460] Webserver started at http://127.0.98.62:41817/ using document root <none> and password file <none>
I20260924 17:42:37.097718 20028 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:37.097787 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:37.097932 20028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:37.098984 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "f505981fd0a243918435bd5ed7e0b5fa"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "b5faaef416274ad0355143107beb39ba"
server_key_iv: "4db7169802e2d4eb3d3636043a57b777"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.099402 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "f505981fd0a243918435bd5ed7e0b5fa"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "b5faaef416274ad0355143107beb39ba"
server_key_iv: "4db7169802e2d4eb3d3636043a57b777"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.100682 20028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:37.101366 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.101548 20028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:37.101645 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f505981fd0a243918435bd5ed7e0b5fa"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "b5faaef416274ad0355143107beb39ba"
server_key_iv: "4db7169802e2d4eb3d3636043a57b777"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.101727 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:37.131525 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:37.131871 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:37.132033 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:37.136039 20028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39409
I20260924 17:42:37.136106 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39409 every 8 connection(s)
I20260924 17:42:37.136479 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:37.137449 20095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:37.139894 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: Bootstrap starting.
I20260924 17:42:37.140527 20095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:37.140766 20095 log.cc:824] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:37.141489 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: No bootstrap required, opened a new log
I20260924 17:42:37.143271 20095 raft_consensus.cc:348] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260924 17:42:37.143426 20095 raft_consensus.cc:374] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:37.143450 20095 raft_consensus.cc:729] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f505981fd0a243918435bd5ed7e0b5fa, State: Initialized, Role: FOLLOWER
I20260924 17:42:37.143596 20095 consensus_queue.cc:260] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260924 17:42:37.143675 20095 raft_consensus.cc:388] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:37.143698 20095 raft_consensus.cc:482] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:37.143726 20095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:37.144208 20095 raft_consensus.cc:504] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260924 17:42:37.144294 20095 leader_election.cc:302] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f505981fd0a243918435bd5ed7e0b5fa; no voters:
I20260924 17:42:37.144428 20095 leader_election.cc:288] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:37.144627 20098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:37.144670 20095 sys_catalog.cc:564] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:37.144819 20098 raft_consensus.cc:686] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [term 1 LEADER]: Becoming Leader. State: Replica: f505981fd0a243918435bd5ed7e0b5fa, State: Running, Role: LEADER
I20260924 17:42:37.144946 20098 consensus_queue.cc:237] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260924 17:42:37.145356 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f505981fd0a243918435bd5ed7e0b5fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } } }
I20260924 17:42:37.145435 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:37.145455 20100 sys_catalog.cc:455] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f505981fd0a243918435bd5ed7e0b5fa. Latest consensus state: current_term: 1 leader_uuid: "f505981fd0a243918435bd5ed7e0b5fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f505981fd0a243918435bd5ed7e0b5fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } } }
I20260924 17:42:37.145545 20100 sys_catalog.cc:458] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:37.145648 20107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:37.146135 20107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:37.146433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20028
I20260924 17:42:37.146548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:37.146744 392 external_mini_cluster.cc:1428] Setting key 9fd084de3c0d60fa1f7b693a51c11390
I20260924 17:42:37.151691 20107 catalog_manager.cc:1347] Generated new cluster ID: 37e223d16a0f41949f7c4a5ee4197007
I20260924 17:42:37.151805 20107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:37.156649 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.147374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.0.98.62:39409)
0924 17:42:37.151299 (+ 3925us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.151309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.151346 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:42:37.151477 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.151488 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.151563 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.151718 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:37.151973 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.153349 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.154461 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.155098 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.155209 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.155926 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:37.155930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:37.155930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:37.156141 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:37.156181 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:37.156194 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:37.156254 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.156337 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.156411 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":3812,"thread_start_us":75,"threads_started":1}
I20260924 17:42:37.169113 20107 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:37.169665 20107 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:37.174926 20107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: Generated new TSK 0
I20260924 17:42:37.175149 20107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:37.176461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--builtin_ntp_servers=127.0.98.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:37.266728 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:37.267077 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:37.267148 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:37.268654 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:37.268725 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:37.268770 20119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 17:42:37.270134 20119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:37.270422 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:37.270742 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:37.276836 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.277151 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.277568 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:37.278267 20119 server_base.cc:1034] running on GCE node
I20260924 17:42:37.278496 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:37.278744 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:37.279937 20119 hybrid_clock.cc:648] HybridClock initialized: now 1790271757279900 us; error 51 us; skew 500 ppm
I20260924 17:42:37.281472 20119 webserver.cc:460] Webserver started at http://127.0.98.1:35907/ using document root <none> and password file <none>
I20260924 17:42:37.281733 20119 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:37.281790 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:37.281956 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:37.283461 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a34528ecbf44d7e9e92e3ae19c384d0"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "c548bb66778ae3273e79f77bb4b58ac3"
server_key_iv: "e888bac0a30a499d044630c4c6454a76"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.283850 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a34528ecbf44d7e9e92e3ae19c384d0"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "c548bb66778ae3273e79f77bb4b58ac3"
server_key_iv: "e888bac0a30a499d044630c4c6454a76"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.285429 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:37.286303 20133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.286489 20119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:37.286634 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal
uuid: "2a34528ecbf44d7e9e92e3ae19c384d0"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "c548bb66778ae3273e79f77bb4b58ac3"
server_key_iv: "e888bac0a30a499d044630c4c6454a76"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.286767 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:37.298019 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:37.298348 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:37.298606 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:37.298892 20119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:37.299306 20119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:37.299398 20119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.299470 20119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:37.299503 20119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.305907 20119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33211
I20260924 17:42:37.305950 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33211 every 8 connection(s)
I20260924 17:42:37.306425 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/data/info.pb
I20260924 17:42:37.313162 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.307298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40417 (local address 127.0.98.62:39409)
0924 17:42:37.307354 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.307356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.307649 (+ 293us) server_negotiation.cc:407] Connection header received
0924 17:42:37.308429 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.308430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.308444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.308503 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:37.309489 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.310011 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.311637 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.311833 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.311892 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.312895 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:37.312896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:37.312896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:37.312902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:37.312916 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:37.312919 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:37.312972 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.313068 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.313097 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:42:37.313467 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.307273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.1:40417)
0924 17:42:37.307491 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.307541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.308281 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.308606 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.308611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.308622 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:37.309350 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.309357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.310108 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.310109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.311551 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.311554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.311928 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.311930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.312103 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.312724 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:37.312724 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:37.312767 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:37.312801 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:37.312983 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:37.312987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:37.312991 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.313067 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.313190 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260924 17:42:37.314076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20119
I20260924 17:42:37.314109 20247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260924 17:42:37.314204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-0/wal/instance
I20260924 17:42:37.314213 20247 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:37.314414 392 external_mini_cluster.cc:1428] Setting key ef62914c5da0c90d1453dd519e9fa0e9
I20260924 17:42:37.314425 20247 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260924 17:42:37.314998 20058 ts_manager.cc:194] Registered new tserver with Master: 2a34528ecbf44d7e9e92e3ae19c384d0 (127.0.98.1:33211)
I20260924 17:42:37.315536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--builtin_ntp_servers=127.0.98.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:42:37.316032 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40417
W20260924 17:42:37.397066 20250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:37.397267 20250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:37.397289 20250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:37.398465 20250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:37.398507 20250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:37.398532 20250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 17:42:37.400007 20250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:37.400249 20250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:37.400547 20250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:37.405957 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.405975 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.405969 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:37.406307 20250 server_base.cc:1034] running on GCE node
I20260924 17:42:37.406582 20250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:37.406838 20250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:37.408030 20250 hybrid_clock.cc:648] HybridClock initialized: now 1790271757407999 us; error 60 us; skew 500 ppm
I20260924 17:42:37.409337 20250 webserver.cc:460] Webserver started at http://127.0.98.2:32839/ using document root <none> and password file <none>
I20260924 17:42:37.409612 20250 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:37.409686 20250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:37.409824 20250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:37.410892 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data/instance:
uuid: "26678ace645a4430bfd3f3a272603d66"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "66b6c707d8f8c02c5e4b88e0bc703c5d"
server_key_iv: "1f9b7157ccf357751c423e4aac4137db"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.411278 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26678ace645a4430bfd3f3a272603d66"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "66b6c707d8f8c02c5e4b88e0bc703c5d"
server_key_iv: "1f9b7157ccf357751c423e4aac4137db"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.412566 20250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:37.413254 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.413437 20250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:42:37.413538 20250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal
uuid: "26678ace645a4430bfd3f3a272603d66"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "66b6c707d8f8c02c5e4b88e0bc703c5d"
server_key_iv: "1f9b7157ccf357751c423e4aac4137db"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.413610 20250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:37.437425 20250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:37.437786 20250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:37.437953 20250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:37.438201 20250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:37.438550 20250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:37.438606 20250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.438648 20250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:37.438694 20250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.444307 20250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32927
I20260924 17:42:37.444386 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32927 every 8 connection(s)
I20260924 17:42:37.444730 20250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/data/info.pb
I20260924 17:42:37.449847 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.445558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52703 (local address 127.0.98.62:39409)
0924 17:42:37.445629 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.445631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.445859 (+ 228us) server_negotiation.cc:407] Connection header received
0924 17:42:37.446331 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.446332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.446343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.446434 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:37.447178 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.447670 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.448645 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.448822 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.448872 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.449616 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:37.449617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:37.449618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:37.449622 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:37.449635 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:37.449639 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:37.449681 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.449764 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.449787 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:42:37.450043 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.445529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.2:52703)
0924 17:42:37.445725 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.445767 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.446246 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.446519 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.446523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.446532 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:37.447085 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.447092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.447752 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.447753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.448568 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.448570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.448896 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.448897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.449004 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.449489 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:37.449489 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:37.449517 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:37.449540 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:37.449695 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:37.449699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:37.449701 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.449753 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.449832 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260924 17:42:37.450505 20378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260924 17:42:37.450587 20378 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:37.450774 20378 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260924 17:42:37.450769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20250
I20260924 17:42:37.450871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal/instance
I20260924 17:42:37.451092 392 external_mini_cluster.cc:1428] Setting key 4c9ced2df2d2ea067461a2ca965a1677
I20260924 17:42:37.451296 20058 ts_manager.cc:194] Registered new tserver with Master: 26678ace645a4430bfd3f3a272603d66 (127.0.98.2:32927)
I20260924 17:42:37.451713 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52703
I20260924 17:42:37.451980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--builtin_ntp_servers=127.0.98.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:42:37.531622 20381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:37.531828 20381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:37.531850 20381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:37.533131 20381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:37.533177 20381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:37.533203 20381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 17:42:37.534471 20381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:37.534667 20381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:37.534906 20381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:37.539805 20389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.539810 20387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:37.539806 20386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:37.540076 20381 server_base.cc:1034] running on GCE node
I20260924 17:42:37.540280 20381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:37.540527 20381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:37.541674 20381 hybrid_clock.cc:648] HybridClock initialized: now 1790271757541665 us; error 27 us; skew 500 ppm
I20260924 17:42:37.542970 20381 webserver.cc:460] Webserver started at http://127.0.98.3:41903/ using document root <none> and password file <none>
I20260924 17:42:37.543248 20381 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:37.543298 20381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:37.543458 20381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:37.544590 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data/instance:
uuid: "22c5340d5d024bc78df2b8442c39f625"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "356ec77b57788aa26ad1218ec94a8491"
server_key_iv: "dabbde961c5dccfcc31b674adddd0d6e"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.545001 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22c5340d5d024bc78df2b8442c39f625"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "356ec77b57788aa26ad1218ec94a8491"
server_key_iv: "dabbde961c5dccfcc31b674adddd0d6e"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.546417 20381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:37.547124 20395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.547274 20381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:37.547364 20381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal
uuid: "22c5340d5d024bc78df2b8442c39f625"
format_stamp: "Formatted at 2026-09-24 17:42:37 on dist-test-slave-d0pf"
server_key: "356ec77b57788aa26ad1218ec94a8491"
server_key_iv: "dabbde961c5dccfcc31b674adddd0d6e"
server_key_version: "encryptionkey@0"
I20260924 17:42:37.547417 20381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:37.570284 20381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:37.570569 20381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:37.570676 20381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:37.570876 20381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:42:37.571249 20381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:42:37.571286 20381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.571310 20381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:42:37.571322 20381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:37.577001 20381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44695
I20260924 17:42:37.577086 20508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44695 every 8 connection(s)
I20260924 17:42:37.577426 20381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/data/info.pb
I20260924 17:42:37.582690 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.578199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50049 (local address 127.0.98.62:39409)
0924 17:42:37.578272 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.578275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.578515 (+ 240us) server_negotiation.cc:407] Connection header received
0924 17:42:37.578982 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.578983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.578993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.579126 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:37.579798 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.580289 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.581295 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.581481 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.581531 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.582476 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:37.582477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:37.582477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:37.582482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:37.582493 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:37.582497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:37.582534 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.582605 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.582627 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:42:37.582926 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.578161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.3:50049)
0924 17:42:37.578385 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.578426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.578903 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.579197 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.579201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.579210 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:42:37.579718 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.579723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.580392 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.580393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.581196 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.581198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.581590 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.581591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.581717 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.582343 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:42:37.582343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:42:37.582375 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:42:37.582399 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:42:37.582536 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:42:37.582539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:42:37.582541 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.582588 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.582669 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260924 17:42:37.583493 20509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260924 17:42:37.583598 20509 heartbeater.cc:464] Registering TS with master...
I20260924 17:42:37.583782 20509 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260924 17:42:37.584173 20058 ts_manager.cc:194] Registered new tserver with Master: 22c5340d5d024bc78df2b8442c39f625 (127.0.98.3:44695)
I20260924 17:42:37.584542 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50049
I20260924 17:42:37.586831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20381
I20260924 17:42:37.586920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal/instance
I20260924 17:42:37.587142 392 external_mini_cluster.cc:1428] Setting key 1f44ed517d52a08840fb0ba4e360aebb
I20260924 17:42:37.588050 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:42:37.592556 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.589389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.0.98.62:39409)
0924 17:42:37.589496 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.589498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.589598 (+ 100us) server_negotiation.cc:407] Connection header received
0924 17:42:37.589677 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.589679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.589698 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.589769 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:37.589971 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.590467 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.591324 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.591510 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.591584 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.592283 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:37.592285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:37.592285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:37.592291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:37.592307 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:37.592312 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:37.592382 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.592444 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.592468 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 17:42:37.594215 20058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:42:37.595851 20058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:42:37.602152 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.598000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.0.98.2:32927)
0924 17:42:37.598175 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.598179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.598232 (+ 53us) server_negotiation.cc:407] Connection header received
0924 17:42:37.598361 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.598365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.598391 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.598476 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:37.598873 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.599665 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.601008 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.601712 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.601750 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.601816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.602025 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.602064 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260924 17:42:37.602141 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.597921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32927 (local address 127.0.0.1:46410)
0924 17:42:37.598129 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.598173 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.598257 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.598502 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.598506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.598508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:37.598581 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.598583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.599753 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.599754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.600908 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.600910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.601863 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.601864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.601919 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.601928 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.602010 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.602063 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260924 17:42:37.602542 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.597958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33211 (local address 127.0.0.1:50344)
0924 17:42:37.598213 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.598223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.598303 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.598760 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.598761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.598763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:37.598799 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.598800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.600240 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.600241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.601132 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.601134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.602140 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.602141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.602348 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.602353 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.602418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.602456 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260924 17:42:37.602670 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.597958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44695 (local address 127.0.0.1:53424)
0924 17:42:37.598381 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.598390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.598431 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.598858 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.598859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.598862 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:37.598916 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.598918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.600211 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.600212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.601298 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.601300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.602421 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.602422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.602475 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.602478 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.602506 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.602536 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":33,"thread_start_us":111,"threads_started":1}
I20260924 17:42:37.602847 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.598206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.0.98.3:44695)
0924 17:42:37.598603 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.598724 (+ 121us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.598734 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:42:37.598772 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.598774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.598788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.598858 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:37.598972 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.600116 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.601382 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.602324 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.602581 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.602688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.602714 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.602752 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":86,"threads_started":1}
I20260924 17:42:37.602928 20312 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 (DEFAULT_TABLE table=test-table [id=6d07a3e64ecd4732b8e71bbaadb18795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:37.603330 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.598387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.98.1:33211)
0924 17:42:37.598581 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.598585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.598595 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:42:37.598671 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.598673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.598687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.598766 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:37.599072 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.600154 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.601215 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.601944 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.602017 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.602103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.603190 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.603256 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260924 17:42:37.603387 20312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2a5c9a4b624236ac5297a9770a43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:37.603524 20443 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 (DEFAULT_TABLE table=test-table [id=6d07a3e64ecd4732b8e71bbaadb18795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:37.603840 20443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2a5c9a4b624236ac5297a9770a43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:37.606895 20523 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Bootstrap starting.
I20260924 17:42:37.607479 20524 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Bootstrap starting.
I20260924 17:42:37.607724 20181 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 (DEFAULT_TABLE table=test-table [id=6d07a3e64ecd4732b8e71bbaadb18795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:42:37.608039 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2a5c9a4b624236ac5297a9770a43b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:37.608063 20524 tablet_bootstrap.cc:654] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:37.608345 20524 log.cc:824] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:37.609002 20524 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: No bootstrap required, opened a new log
I20260924 17:42:37.609055 20524 ts_tablet_manager.cc:1397] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:37.610858 20526 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Bootstrap starting.
I20260924 17:42:37.611280 20523 tablet_bootstrap.cc:654] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:37.611351 20524 raft_consensus.cc:348] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.611524 20526 tablet_bootstrap.cc:654] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:37.611526 20524 raft_consensus.cc:374] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:37.611562 20524 raft_consensus.cc:729] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c5340d5d024bc78df2b8442c39f625, State: Initialized, Role: FOLLOWER
I20260924 17:42:37.611630 20523 log.cc:824] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:37.611714 20524 consensus_queue.cc:260] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.611832 20526 log.cc:824] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:37.611987 20524 ts_tablet_manager.cc:1428] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:37.612099 20509 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
I20260924 17:42:37.612476 20523 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: No bootstrap required, opened a new log
I20260924 17:42:37.612527 20523 ts_tablet_manager.cc:1397] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 17:42:37.612732 20526 tablet_bootstrap.cc:492] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: No bootstrap required, opened a new log
I20260924 17:42:37.612794 20526 ts_tablet_manager.cc:1397] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:42:37.614491 20523 raft_consensus.cc:348] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.614648 20523 raft_consensus.cc:374] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:37.614698 20523 raft_consensus.cc:729] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26678ace645a4430bfd3f3a272603d66, State: Initialized, Role: FOLLOWER
I20260924 17:42:37.614847 20523 consensus_queue.cc:260] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.614817 20526 raft_consensus.cc:348] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.614948 20526 raft_consensus.cc:374] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:37.615008 20526 raft_consensus.cc:729] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a34528ecbf44d7e9e92e3ae19c384d0, State: Initialized, Role: FOLLOWER
I20260924 17:42:37.615140 20523 ts_tablet_manager.cc:1428] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:37.615150 20526 consensus_queue.cc:260] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.615283 20378 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
I20260924 17:42:37.615423 20526 ts_tablet_manager.cc:1428] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:42:37.615533 20247 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
W20260924 17:42:37.695295 20379 tablet.cc:2369] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:42:37.807185 20248 tablet.cc:2369] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:42:37.828096 20510 tablet.cc:2369] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:42:37.980237 20531 raft_consensus.cc:482] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:42:37.980371 20531 raft_consensus.cc:504] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.980715 20531 leader_election.cc:288] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26678ace645a4430bfd3f3a272603d66 (127.0.98.2:32927), 22c5340d5d024bc78df2b8442c39f625 (127.0.98.3:44695)
I20260924 17:42:37.983660 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.980955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34285 (local address 127.0.98.2:32927)
0924 17:42:37.981010 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.981013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.981129 (+ 116us) server_negotiation.cc:407] Connection header received
0924 17:42:37.981200 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.981201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.981214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.981285 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:37.981473 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.981947 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.982918 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.983301 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.983355 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.983362 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.983535 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.983582 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:42:37.983670 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.980904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32927 (local address 127.0.98.1:34285)
0924 17:42:37.981039 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.981050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.981118 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.981306 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.981307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.981309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:37.981379 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.981380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.982061 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.982062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.982802 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.982804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.983380 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.983382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.983455 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.983459 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.983509 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.983559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":10,"thread_start_us":62,"threads_started":1}
I20260924 17:42:37.983703 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.981021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46745 (local address 127.0.98.3:44695)
0924 17:42:37.981068 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:37.981070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:37.981080 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:42:37.981185 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:37.981186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:37.981197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:37.981249 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:42:37.981432 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.981926 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.982882 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.983321 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.983406 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.983415 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:37.983587 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 17:42:37.983625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:42:37.983685 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:37.980887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44695 (local address 127.0.98.1:46745)
0924 17:42:37.980998 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:37.981010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:42:37.981075 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:42:37.981276 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:42:37.981277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:42:37.981279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:42:37.981358 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.981360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.982009 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.982010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.982787 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:42:37.982789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:42:37.983435 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:42:37.983436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:42:37.983488 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:37.983509 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 17:42:37.983566 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:42:37.983599 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260924 17:42:37.983989 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2a5c9a4b624236ac5297a9770a43b8" candidate_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26678ace645a4430bfd3f3a272603d66" is_pre_election: true
I20260924 17:42:37.984025 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2a5c9a4b624236ac5297a9770a43b8" candidate_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c5340d5d024bc78df2b8442c39f625" is_pre_election: true
I20260924 17:42:37.984153 20463 raft_consensus.cc:2454] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a34528ecbf44d7e9e92e3ae19c384d0 in term 0.
I20260924 17:42:37.984153 20332 raft_consensus.cc:2454] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a34528ecbf44d7e9e92e3ae19c384d0 in term 0.
I20260924 17:42:37.984383 20136 leader_election.cc:302] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22c5340d5d024bc78df2b8442c39f625, 26678ace645a4430bfd3f3a272603d66, 2a34528ecbf44d7e9e92e3ae19c384d0; no voters:
I20260924 17:42:37.984516 20531 raft_consensus.cc:2781] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:42:37.984571 20531 raft_consensus.cc:482] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:42:37.984596 20531 raft_consensus.cc:3037] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:37.985266 20531 raft_consensus.cc:504] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.985405 20531 leader_election.cc:288] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [CANDIDATE]: Term 1 election: Requested vote from peers 26678ace645a4430bfd3f3a272603d66 (127.0.98.2:32927), 22c5340d5d024bc78df2b8442c39f625 (127.0.98.3:44695)
I20260924 17:42:37.985564 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2a5c9a4b624236ac5297a9770a43b8" candidate_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c5340d5d024bc78df2b8442c39f625"
I20260924 17:42:37.985576 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2a5c9a4b624236ac5297a9770a43b8" candidate_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26678ace645a4430bfd3f3a272603d66"
I20260924 17:42:37.985677 20332 raft_consensus.cc:3037] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:37.985658 20463 raft_consensus.cc:3037] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:37.986393 20332 raft_consensus.cc:2454] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a34528ecbf44d7e9e92e3ae19c384d0 in term 1.
I20260924 17:42:37.986406 20463 raft_consensus.cc:2454] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a34528ecbf44d7e9e92e3ae19c384d0 in term 1.
I20260924 17:42:37.986614 20137 leader_election.cc:302] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26678ace645a4430bfd3f3a272603d66, 2a34528ecbf44d7e9e92e3ae19c384d0; no voters:
I20260924 17:42:37.986722 20531 raft_consensus.cc:2781] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:37.986871 20531 raft_consensus.cc:686] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a34528ecbf44d7e9e92e3ae19c384d0, State: Running, Role: LEADER
I20260924 17:42:37.986968 20531 consensus_queue.cc:237] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } }
I20260924 17:42:37.987838 20058 catalog_manager.cc:5495] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a34528ecbf44d7e9e92e3ae19c384d0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a34528ecbf44d7e9e92e3ae19c384d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33211 } health_report { overall_health: HEALTHY } } }
I20260924 17:42:38.047700 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:38.045351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.98.1:33211)
0924 17:42:38.045439 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:38.045441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:38.045450 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:42:38.045504 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:38.045505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:38.045525 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:38.045583 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:42:38.045739 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:38.046238 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:38.046966 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:38.047234 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:38.047289 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:38.047413 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:42:38.047505 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:42:38.047554 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:38.047570 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 17:42:38.047586 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:42:38.049590 20332 raft_consensus.cc:1261] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 1 FOLLOWER]: Refusing update from remote peer 2a34528ecbf44d7e9e92e3ae19c384d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:38.049594 20463 raft_consensus.cc:1261] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 1 FOLLOWER]: Refusing update from remote peer 2a34528ecbf44d7e9e92e3ae19c384d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:42:38.049983 20534 consensus_queue.cc:1035] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c5340d5d024bc78df2b8442c39f625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:38.050140 20531 consensus_queue.cc:1035] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26678ace645a4430bfd3f3a272603d66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:42:38.052518 20537 mvcc.cc:205] Tried to move back new op lower bound from 7332953120968970240 to 7332953120715567104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953120968970240}]
I20260924 17:42:38.052935 20539 mvcc.cc:205] Tried to move back new op lower bound from 7332953120968970240 to 7332953120715567104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953120968970240}]
I20260924 17:42:38.053426 20538 mvcc.cc:205] Tried to move back new op lower bound from 7332953120968970240 to 7332953120715567104. Current Snapshot: MvccSnapshot[applied={T|T < 7332953120968970240 or (T in {7332953120968970240})}]
I20260924 17:42:38.054826 20058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:38.054926 20058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:42:38.055763 20058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f505981fd0a243918435bd5ed7e0b5fa: Sending DeleteTablet for 3 replicas of tablet 5c2a5c9a4b624236ac5297a9770a43b8
I20260924 17:42:38.056077 20181 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:38 UTC) from {username='slave'} at 127.0.0.1:50344
I20260924 17:42:38.056063 20312 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:38 UTC) from {username='slave'} at 127.0.0.1:46410
I20260924 17:42:38.056155 20443 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2a5c9a4b624236ac5297a9770a43b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:42:38 UTC) from {username='slave'} at 127.0.0.1:53424
I20260924 17:42:38.056427 20547 tablet_replica.cc:330] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0: stopping tablet replica
I20260924 17:42:38.056469 20548 tablet_replica.cc:330] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: stopping tablet replica
I20260924 17:42:38.056568 20548 raft_consensus.cc:2229] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:38.056586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20119
I20260924 17:42:38.056674 20547 raft_consensus.cc:2229] T 5c2a5c9a4b624236ac5297a9770a43b8 P 2a34528ecbf44d7e9e92e3ae19c384d0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:42:38.056692 20548 raft_consensus.cc:2258] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:38.057128 20548 ts_tablet_manager.cc:1905] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:38.058591 20549 tablet_replica.cc:330] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: stopping tablet replica
I20260924 17:42:38.058683 20549 raft_consensus.cc:2229] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:42:38.058763 20549 raft_consensus.cc:2258] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:42:38.059103 20548 ts_tablet_manager.cc:1918] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:38.059181 20548 log.cc:1196] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-1/wal/wals/5c2a5c9a4b624236ac5297a9770a43b8
I20260924 17:42:38.059254 20549 ts_tablet_manager.cc:1905] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:42:38.059458 20548 ts_tablet_manager.cc:1939] T 5c2a5c9a4b624236ac5297a9770a43b8 P 26678ace645a4430bfd3f3a272603d66: Deleting consensus metadata
I20260924 17:42:38.059950 20046 catalog_manager.cc:4841] TS 26678ace645a4430bfd3f3a272603d66 (127.0.98.2:32927): tablet 5c2a5c9a4b624236ac5297a9770a43b8 (table test-table [id=6d07a3e64ecd4732b8e71bbaadb18795]) successfully deleted
I20260924 17:42:38.060494 20549 ts_tablet_manager.cc:1918] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:42:38.060554 20549 log.cc:1196] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Deleting WAL directory at /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271516406493-392-0/minicluster-data/ts-2/wal/wals/5c2a5c9a4b624236ac5297a9770a43b8
I20260924 17:42:38.060791 20549 ts_tablet_manager.cc:1939] T 5c2a5c9a4b624236ac5297a9770a43b8 P 22c5340d5d024bc78df2b8442c39f625: Deleting consensus metadata
I20260924 17:42:38.061264 20045 catalog_manager.cc:4841] TS 22c5340d5d024bc78df2b8442c39f625 (127.0.98.3:44695): tablet 5c2a5c9a4b624236ac5297a9770a43b8 (table test-table [id=6d07a3e64ecd4732b8e71bbaadb18795]) successfully deleted
W20260924 17:42:38.062688 20044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33211 (error 108)
I20260924 17:42:38.063242 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:38.063135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33211 (local address 127.0.0.1:50374)
0924 17:42:38.063178 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 17:42:38.063215 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 17:42:38.063328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20250
W20260924 17:42:38.063391 20044 catalog_manager.cc:4584] TS 2a34528ecbf44d7e9e92e3ae19c384d0 (127.0.98.1:33211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c2a5c9a4b624236ac5297a9770a43b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33211: connect: Connection refused (error 111)
I20260924 17:42:38.069065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20381
I20260924 17:42:38.073926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20028
2026-09-24T17:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1099 ms)
[----------] 6 tests from EncryptionPolicyTest (8414 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:38 dist-test-slave-d0pf krb5kdc[20553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:38 dist-test-slave-d0pf krb5kdc[20553](info): set up 2 sockets
Sep 24 17:42:38 dist-test-slave-d0pf krb5kdc[20553](info): commencing operation
krb5kdc: starting...
W20260924 17:42:39.111145 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:39.234263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:39.320688 20566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:39.320961 20566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:39.321003 20566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:39.322270 20566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:39.322350 20566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:39.322377 20566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:39.322400 20566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:39.322429 20566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:39.324079 20566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:39.324393 20566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:39.324641 20566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:39.329628 20571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:39.329628 20572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:39.329628 20574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:39.329783 20566 server_base.cc:1034] running on GCE node
I20260924 17:42:39.330345 20566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:39.330633 20566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:39.331795 20566 hybrid_clock.cc:648] HybridClock initialized: now 1790271759331773 us; error 37 us; skew 500 ppm
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:39.334476 20566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:39.334969 20566 webserver.cc:460] Webserver started at http://127.0.0.1:44837/ using document root <none> and password file <none>
I20260924 17:42:39.335265 20566 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:39.335342 20566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:39.335495 20566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:39.336736 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "9e122b48e9f84dd89276100fe7b99f08"
format_stamp: "Formatted at 2026-09-24 17:42:39 on dist-test-slave-d0pf"
server_key: "dd044c419a8bfe45dda55e78d24dd8a9"
server_key_iv: "6bc36c8ca4e1b676029ddfc9a0d0b860"
server_key_version: "encryptionkey@0"
I20260924 17:42:39.337112 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e122b48e9f84dd89276100fe7b99f08"
format_stamp: "Formatted at 2026-09-24 17:42:39 on dist-test-slave-d0pf"
server_key: "dd044c419a8bfe45dda55e78d24dd8a9"
server_key_iv: "6bc36c8ca4e1b676029ddfc9a0d0b860"
server_key_version: "encryptionkey@0"
I20260924 17:42:39.338547 20566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:39.339447 20581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:39.339638 20566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:42:39.339736 20566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "9e122b48e9f84dd89276100fe7b99f08"
format_stamp: "Formatted at 2026-09-24 17:42:39 on dist-test-slave-d0pf"
server_key: "dd044c419a8bfe45dda55e78d24dd8a9"
server_key_iv: "6bc36c8ca4e1b676029ddfc9a0d0b860"
server_key_version: "encryptionkey@0"
I20260924 17:42:39.339826 20566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:39.358513 20566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:39.361053 20566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:39.361238 20566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:39.365330 20566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43561
I20260924 17:42:39.365423 20633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43561 every 8 connection(s)
I20260924 17:42:39.365773 20566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:39.366693 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:39.369159 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08: Bootstrap starting.
I20260924 17:42:39.369719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20566
I20260924 17:42:39.369810 20634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:39.369823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:39.370038 392 external_mini_cluster.cc:1428] Setting key f72e666bb0a1d46ff78f7452f867f283
I20260924 17:42:39.370136 20634 log.cc:824] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:39.371170 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08: No bootstrap required, opened a new log
I20260924 17:42:39.373230 20634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260924 17:42:39.373378 20634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:39.373400 20634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e122b48e9f84dd89276100fe7b99f08, State: Initialized, Role: FOLLOWER
I20260924 17:42:39.373545 20634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260924 17:42:39.373656 20634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:39.373703 20634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:39.373765 20634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:39.374358 20634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260924 17:42:39.374485 20634 leader_election.cc:302] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e122b48e9f84dd89276100fe7b99f08; no voters:
I20260924 17:42:39.374702 20634 leader_election.cc:288] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:39.374732 20639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:39.374854 20639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [term 1 LEADER]: Becoming Leader. State: Replica: 9e122b48e9f84dd89276100fe7b99f08, State: Running, Role: LEADER
I20260924 17:42:39.374989 20634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:39.375003 20639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260924 17:42:39.375517 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e122b48e9f84dd89276100fe7b99f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } } }
I20260924 17:42:39.375615 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:39.375803 20641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e122b48e9f84dd89276100fe7b99f08. Latest consensus state: current_term: 1 leader_uuid: "9e122b48e9f84dd89276100fe7b99f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e122b48e9f84dd89276100fe7b99f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } } }
I20260924 17:42:39.375880 20641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:39.376045 20648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:39.376618 20648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:42:39.379774 20648 catalog_manager.cc:1347] Generated new cluster ID: bfe7c0df6ad644e0a0129b24bde0d8b0
I20260924 17:42:39.379863 20648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:39.382678 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:39.370621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45063 (local address 127.0.0.1:43561)
0924 17:42:39.370963 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:39.370971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:39.370998 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:42:39.371577 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:39.371585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:39.371639 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:39.371806 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:39.372473 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:39.373433 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:39.374315 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:39.374911 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:39.375338 (+ 427us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:39.378588 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:39.378593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:39.378593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:39.378607 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:39.380721 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:39.381236 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:39.381239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:39.381240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:39.381344 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:39.381598 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:39.381599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:39.381599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:39.381900 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:39.382055 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:39.382203 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 17:42:39.382308 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":218,"threads_started":1}
I20260924 17:42:39.383677 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:39.392882 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:39.385133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44397 (local address 127.0.0.1:43561)
0924 17:42:39.385229 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:39.385231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:39.385295 (+ 64us) server_negotiation.cc:407] Connection header received
0924 17:42:39.385369 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:39.385370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:39.385379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:39.385427 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:39.386237 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:39.386994 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:39.388073 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:39.388387 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:39.388448 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:39.389974 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:39.389975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:39.389975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:39.389981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:39.391527 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:39.391955 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:39.391956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:39.391957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:39.392012 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:39.392235 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:39.392236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:39.392236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:39.392354 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:39.392447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:39.392593 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 17:42:39.392651 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 17:42:39.394781 20648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:39.395365 20648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:39.399782 20648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e122b48e9f84dd89276100fe7b99f08: Generated new TSK 0
I20260924 17:42:39.400002 20648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:39.404459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20566
W20260924 17:42:39.409139 20551 connection.cc:535] client connection to 127.0.0.1:43561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43561): Connection reset by peer (error 104)
2026-09-24T17:42:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20665](info): set up 2 sockets
Sep 24 17:42:39 dist-test-slave-d0pf krb5kdc[20665](info): commencing operation
krb5kdc: starting...
W20260924 17:42:40.450984 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:40 dist-test-slave-d0pf krb5kdc[20665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:40.573735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:40.653625 20678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:40.653842 20678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:40.653865 20678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:40.655097 20678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:40.655145 20678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:40.655160 20678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:40.655172 20678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:40.655185 20678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:40.656697 20678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:40.656961 20678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:40.657195 20678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:40.662102 20684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:40.662077 20686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:40.662161 20683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:40.662257 20678 server_base.cc:1034] running on GCE node
I20260924 17:42:40.662638 20678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:40.662910 20678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:40.664057 20678 hybrid_clock.cc:648] HybridClock initialized: now 1790271760664029 us; error 40 us; skew 500 ppm
Sep 24 17:42:40 dist-test-slave-d0pf krb5kdc[20665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:40.673002 20678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:40.673467 20678 webserver.cc:460] Webserver started at http://127.0.0.1:41589/ using document root <none> and password file <none>
I20260924 17:42:40.673702 20678 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:40.673748 20678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:40.673916 20678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:40.675004 20678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "9a979bdd29c440a1855e3b0f0575781a"
format_stamp: "Formatted at 2026-09-24 17:42:40 on dist-test-slave-d0pf"
server_key: "9bb46ecb169e1231f332b4c1da399ef2"
server_key_iv: "42e37b13116abf852cfce346ac5e12b9"
server_key_version: "encryptionkey@0"
I20260924 17:42:40.675431 20678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a979bdd29c440a1855e3b0f0575781a"
format_stamp: "Formatted at 2026-09-24 17:42:40 on dist-test-slave-d0pf"
server_key: "9bb46ecb169e1231f332b4c1da399ef2"
server_key_iv: "42e37b13116abf852cfce346ac5e12b9"
server_key_version: "encryptionkey@0"
I20260924 17:42:40.676779 20678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:40.677603 20693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:40.677815 20678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:40.677913 20678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "9a979bdd29c440a1855e3b0f0575781a"
format_stamp: "Formatted at 2026-09-24 17:42:40 on dist-test-slave-d0pf"
server_key: "9bb46ecb169e1231f332b4c1da399ef2"
server_key_iv: "42e37b13116abf852cfce346ac5e12b9"
server_key_version: "encryptionkey@0"
I20260924 17:42:40.678017 20678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:40.696328 20678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:40.698513 20678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:40.698666 20678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:40.702790 20678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33091
I20260924 17:42:40.702821 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33091 every 8 connection(s)
I20260924 17:42:40.703229 20678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:40.704078 20746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:40.706302 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a: Bootstrap starting.
I20260924 17:42:40.706887 20746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:40.707203 20746 log.cc:824] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:40.707934 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a: No bootstrap required, opened a new log
I20260924 17:42:40.708736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20678
I20260924 17:42:40.708879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:40.709084 392 external_mini_cluster.cc:1428] Setting key b19e44e13cb4381bd9189eebf013b4d8
I20260924 17:42:40.710222 20746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260924 17:42:40.710394 20746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:40.710433 20746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a979bdd29c440a1855e3b0f0575781a, State: Initialized, Role: FOLLOWER
I20260924 17:42:40.710631 20746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260924 17:42:40.710726 20746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:40.710758 20746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:40.710816 20746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:40.711575 20746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260924 17:42:40.711681 20746 leader_election.cc:302] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a979bdd29c440a1855e3b0f0575781a; no voters:
I20260924 17:42:40.711835 20746 leader_election.cc:288] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:40.711895 20751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:40.712060 20746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:40.712087 20751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [term 1 LEADER]: Becoming Leader. State: Replica: 9a979bdd29c440a1855e3b0f0575781a, State: Running, Role: LEADER
I20260924 17:42:40.712273 20751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260924 17:42:40.712651 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a979bdd29c440a1855e3b0f0575781a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260924 17:42:40.712728 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:40.712913 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a979bdd29c440a1855e3b0f0575781a. Latest consensus state: current_term: 1 leader_uuid: "9a979bdd29c440a1855e3b0f0575781a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a979bdd29c440a1855e3b0f0575781a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260924 17:42:40.713014 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:40.713479 20760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:40.714336 20760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:40.715888 20760 catalog_manager.cc:1347] Generated new cluster ID: 6a83abff89a842108f3835210d0646d6
I20260924 17:42:40.715940 20760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:40.720108 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:40.709610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54753 (local address 127.0.0.1:33091)
0924 17:42:40.709805 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:40.709814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:40.709847 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:42:40.710185 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:40.710194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:40.710258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:40.710404 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:40.710662 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:40.712357 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:40.713475 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:40.714186 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:40.714309 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:40.719287 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:40.719290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:40.719291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:40.719507 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:40.719543 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:40.719556 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:40.719624 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:40.719866 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 17:42:40.719894 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 17:42:40.721102 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:40.726156 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:40.722650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36013 (local address 127.0.0.1:33091)
0924 17:42:40.722688 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:40.722690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:40.722763 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:42:40.722846 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:40.722846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:40.722864 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:40.722909 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:40.723084 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:40.723794 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:40.724811 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:40.725121 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:40.725181 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:40.725910 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:40.725911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:40.725911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:40.725916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:40.725929 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:40.725933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:40.725985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:40.726068 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 17:42:40.726080 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:42:40.737327 20760 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:40.738296 20760 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:40.746765 20760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a979bdd29c440a1855e3b0f0575781a: Generated new TSK 0
I20260924 17:42:40.746954 20760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:40.753209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20678
W20260924 17:42:40.757427 20663 connection.cc:535] client connection to 127.0.0.1:33091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33091): Connection reset by peer (error 104)
2026-09-24T17:42:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:40 dist-test-slave-d0pf krb5kdc[20777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:40 dist-test-slave-d0pf krb5kdc[20777](info): set up 2 sockets
Sep 24 17:42:40 dist-test-slave-d0pf krb5kdc[20777](info): commencing operation
krb5kdc: starting...
W20260924 17:42:41.787120 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:41 dist-test-slave-d0pf krb5kdc[20777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:41.906069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:41.988646 20790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:41.988847 20790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:41.988870 20790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:41.990037 20790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:41.990080 20790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:41.990095 20790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:41.990149 20790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:41.990176 20790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:41.991796 20790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:41.992048 20790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:41.992276 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:41.997220 20795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:41.997308 20790 server_base.cc:1034] running on GCE node
W20260924 17:42:41.997252 20796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:41.997246 20798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:41.997664 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:41.997962 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:41.999122 20790 hybrid_clock.cc:648] HybridClock initialized: now 1790271761999108 us; error 27 us; skew 500 ppm
Sep 24 17:42:41 dist-test-slave-d0pf krb5kdc[20777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:42.001703 20790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:42.002197 20790 webserver.cc:460] Webserver started at http://127.0.0.1:39659/ using document root <none> and password file <none>
I20260924 17:42:42.002451 20790 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:42.002525 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:42.002677 20790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:42.003822 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "1c6c8843096c42a0a4ccb08bdd770d9f"
format_stamp: "Formatted at 2026-09-24 17:42:41 on dist-test-slave-d0pf"
server_key: "ab28dbe6a0d7bac417a10205fa215c28"
server_key_iv: "52f36cb89e17c224904f93ab9e2e09e2"
server_key_version: "encryptionkey@0"
I20260924 17:42:42.004186 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c6c8843096c42a0a4ccb08bdd770d9f"
format_stamp: "Formatted at 2026-09-24 17:42:41 on dist-test-slave-d0pf"
server_key: "ab28dbe6a0d7bac417a10205fa215c28"
server_key_iv: "52f36cb89e17c224904f93ab9e2e09e2"
server_key_version: "encryptionkey@0"
I20260924 17:42:42.005564 20790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:42.006361 20805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:42.006534 20790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:42.006642 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "1c6c8843096c42a0a4ccb08bdd770d9f"
format_stamp: "Formatted at 2026-09-24 17:42:41 on dist-test-slave-d0pf"
server_key: "ab28dbe6a0d7bac417a10205fa215c28"
server_key_iv: "52f36cb89e17c224904f93ab9e2e09e2"
server_key_version: "encryptionkey@0"
I20260924 17:42:42.006722 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:42.010093 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:42.012490 20790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:42.012661 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:42.016749 20790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37165
I20260924 17:42:42.016781 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37165 every 8 connection(s)
I20260924 17:42:42.017127 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:42.018090 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:42.020411 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f: Bootstrap starting.
I20260924 17:42:42.020773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20790
I20260924 17:42:42.020910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:42.021055 20858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:42.021076 392 external_mini_cluster.cc:1428] Setting key 8102f1cc8afd90ee3d8b282fd00b7602
I20260924 17:42:42.021353 20858 log.cc:824] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:42.022037 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f: No bootstrap required, opened a new log
I20260924 17:42:42.023757 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:42.021620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40141 (local address 127.0.0.1:37165)
0924 17:42:42.021803 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:42.021811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:42.021836 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:42:42.022311 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:42.022318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:42.022345 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:42.022491 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:42.023224 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:42.023229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:42.023235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:42.023239 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:42.023272 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:42.023282 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:42.023375 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:42.023414 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 17:42:42.023455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260924 17:42:42.023761 20858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260924 17:42:42.023981 20858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:42.024020 20858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6c8843096c42a0a4ccb08bdd770d9f, State: Initialized, Role: FOLLOWER
I20260924 17:42:42.024139 20858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260924 17:42:42.024230 20858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:42.024294 20858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:42.024335 20858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:42.024683 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:42.024976 20858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260924 17:42:42.025091 20858 leader_election.cc:302] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c6c8843096c42a0a4ccb08bdd770d9f; no voters:
I20260924 17:42:42.025273 20858 leader_election.cc:288] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:42.025319 20863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:42.025486 20863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [term 1 LEADER]: Becoming Leader. State: Replica: 1c6c8843096c42a0a4ccb08bdd770d9f, State: Running, Role: LEADER
I20260924 17:42:42.025519 20858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:42.025640 20863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260924 17:42:42.025985 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } } }
I20260924 17:42:42.026108 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:42.025969 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c6c8843096c42a0a4ccb08bdd770d9f. Latest consensus state: current_term: 1 leader_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c8843096c42a0a4ccb08bdd770d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } } }
I20260924 17:42:42.026312 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:42.026305 20873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:42.026929 20873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:42.027833 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:42.026620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59541 (local address 127.0.0.1:37165)
0924 17:42:42.026661 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:42.026663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:42.026702 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:42:42.026797 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:42.026798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:42.026803 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:42.026831 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:42.027602 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:42.027603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:42.027603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:42.027607 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:42.027619 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:42.027623 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:42.027703 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:42.027782 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 17:42:42.027793 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:42:42.028756 20873 catalog_manager.cc:1347] Generated new cluster ID: e1ad7371ecea47c2a7f4e8eb5445b03d
I20260924 17:42:42.028807 20873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:42.042366 20873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:42.042572 20873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:42.047761 20873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c6c8843096c42a0a4ccb08bdd770d9f: Generated new TSK 0
I20260924 17:42:42.047963 20873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:42.058307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20790
2026-09-24T17:42:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:42 dist-test-slave-d0pf krb5kdc[20889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:42 dist-test-slave-d0pf krb5kdc[20889](info): set up 2 sockets
Sep 24 17:42:42 dist-test-slave-d0pf krb5kdc[20889](info): commencing operation
krb5kdc: starting...
W20260924 17:42:43.099361 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[20889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:43.217456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:43.297861 20902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:43.298206 20902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:43.298247 20902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:43.300056 20902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:43.300129 20902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:43.300154 20902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:43.300184 20902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:43.300213 20902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:43.302069 20902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:43.302356 20902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:43.302591 20902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:43.307471 20910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:43.307480 20907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:43.307494 20902 server_base.cc:1034] running on GCE node
W20260924 17:42:43.307487 20908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:43.307932 20902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:43.308233 20902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:43.309407 20902 hybrid_clock.cc:648] HybridClock initialized: now 1790271763309402 us; error 37 us; skew 500 ppm
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[20889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:43.311844 20902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:43.312265 20902 webserver.cc:460] Webserver started at http://127.0.0.1:34541/ using document root <none> and password file <none>
I20260924 17:42:43.312502 20902 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:43.312568 20902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:43.312716 20902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:43.313748 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "bdeb7eb5c2754418944795e1acc3603a"
format_stamp: "Formatted at 2026-09-24 17:42:43 on dist-test-slave-d0pf"
server_key: "2eccb8ab27052d4de09b451ec83b564e"
server_key_iv: "c9d836261673a685e0235f7e7ce328fb"
server_key_version: "encryptionkey@0"
I20260924 17:42:43.314121 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdeb7eb5c2754418944795e1acc3603a"
format_stamp: "Formatted at 2026-09-24 17:42:43 on dist-test-slave-d0pf"
server_key: "2eccb8ab27052d4de09b451ec83b564e"
server_key_iv: "c9d836261673a685e0235f7e7ce328fb"
server_key_version: "encryptionkey@0"
I20260924 17:42:43.315430 20902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:42:43.316105 20917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:43.316278 20902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:43.316367 20902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "bdeb7eb5c2754418944795e1acc3603a"
format_stamp: "Formatted at 2026-09-24 17:42:43 on dist-test-slave-d0pf"
server_key: "2eccb8ab27052d4de09b451ec83b564e"
server_key_iv: "c9d836261673a685e0235f7e7ce328fb"
server_key_version: "encryptionkey@0"
I20260924 17:42:43.316445 20902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:43.340198 20902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:43.342337 20902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:43.342490 20902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:43.346575 20902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38899
I20260924 17:42:43.346628 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38899 every 8 connection(s)
I20260924 17:42:43.346990 20902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:43.347909 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:43.350126 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a: Bootstrap starting.
I20260924 17:42:43.350695 20970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:43.350987 20970 log.cc:824] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:43.351727 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a: No bootstrap required, opened a new log
I20260924 17:42:43.352388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 20902
I20260924 17:42:43.352499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:43.352669 392 external_mini_cluster.cc:1428] Setting key 04e692810d2f0767cab16f34e2117c64
I20260924 17:42:43.353811 20970 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260924 17:42:43.353974 20970 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:43.354017 20970 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdeb7eb5c2754418944795e1acc3603a, State: Initialized, Role: FOLLOWER
I20260924 17:42:43.354149 20970 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260924 17:42:43.354249 20970 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:43.354293 20970 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:43.354346 20970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:43.355294 20970 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260924 17:42:43.355417 20970 leader_election.cc:302] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdeb7eb5c2754418944795e1acc3603a; no voters:
I20260924 17:42:43.355592 20970 leader_election.cc:288] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:43.355643 20975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:43.355778 20975 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [term 1 LEADER]: Becoming Leader. State: Replica: bdeb7eb5c2754418944795e1acc3603a, State: Running, Role: LEADER
I20260924 17:42:43.355859 20970 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:43.355862 20975 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260924 17:42:43.356245 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdeb7eb5c2754418944795e1acc3603a. Latest consensus state: current_term: 1 leader_uuid: "bdeb7eb5c2754418944795e1acc3603a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } } }
I20260924 17:42:43.356324 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:43.356483 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdeb7eb5c2754418944795e1acc3603a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdeb7eb5c2754418944795e1acc3603a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } } }
I20260924 17:42:43.356541 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:43.356632 20982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:43.357074 20982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:43.359444 20982 catalog_manager.cc:1347] Generated new cluster ID: eb5d3dcb85c84b4a86ee6b3b22b651ac
I20260924 17:42:43.359499 20982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[20889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:42:43.364274 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:43.353194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52715 (local address 127.0.0.1:38899)
0924 17:42:43.353380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:43.353390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:43.353423 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:42:43.353798 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:43.353804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:43.353851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:43.353991 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:43.354721 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:43.355906 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:43.357047 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:43.357815 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:43.357951 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:43.360825 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:43.360829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:43.360829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:43.360842 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:43.362497 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:43.362897 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:43.362900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:43.362901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:43.363017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:43.363272 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:43.363273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:43.363274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:43.363526 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:43.363704 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:43.363802 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 17:42:43.363950 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 17:42:43.365238 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:43.366876 20982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:43.367547 20982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:43.373353 20982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdeb7eb5c2754418944795e1acc3603a: Generated new TSK 0
I20260924 17:42:43.373523 20982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:43.374374 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:43.366672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40811 (local address 127.0.0.1:38899)
0924 17:42:43.366715 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:43.366716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:43.367229 (+ 513us) server_negotiation.cc:407] Connection header received
0924 17:42:43.367290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:43.367291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:43.367301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:43.367360 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:43.367997 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:43.368703 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:43.369723 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:43.370040 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:43.370104 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:43.371595 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:43.371596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:43.371597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:43.371604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:43.373189 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:43.373588 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:43.373589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:43.373590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:43.373645 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:43.373869 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:43.373870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:43.373870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:43.373981 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:43.374086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:43.374193 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:42:43.374281 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 17:42:43.376502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 20902
2026-09-24T17:42:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[21001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[21001](info): set up 2 sockets
Sep 24 17:42:43 dist-test-slave-d0pf krb5kdc[21001](info): commencing operation
krb5kdc: starting...
W20260924 17:42:44.411109 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:44 dist-test-slave-d0pf krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:44.534317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:44.615483 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:44.615739 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:44.615782 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:44.616988 21014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:44.617067 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:44.617095 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:44.617128 21014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:44.617153 21014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:44.618654 21014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:44.618942 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:44.619210 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:44.624537 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:44.624526 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:44.624579 21014 server_base.cc:1034] running on GCE node
W20260924 17:42:44.624521 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:44.624998 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:44.625317 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:44.626485 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790271764626470 us; error 46 us; skew 500 ppm
Sep 24 17:42:44 dist-test-slave-d0pf krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:44.628986 21014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:44.629367 21014 webserver.cc:460] Webserver started at http://127.0.0.1:42585/ using document root <none> and password file <none>
I20260924 17:42:44.629542 21014 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:44.629582 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:44.629675 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:44.630686 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "0097684ec45b462d8471662e85a12b53"
format_stamp: "Formatted at 2026-09-24 17:42:44 on dist-test-slave-d0pf"
server_key: "72e502c541f1ad2a4cbfba20945c778d"
server_key_iv: "4faa3b7aa53f9c644e18f909526d7ae9"
server_key_version: "encryptionkey@0"
I20260924 17:42:44.630983 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "0097684ec45b462d8471662e85a12b53"
format_stamp: "Formatted at 2026-09-24 17:42:44 on dist-test-slave-d0pf"
server_key: "72e502c541f1ad2a4cbfba20945c778d"
server_key_iv: "4faa3b7aa53f9c644e18f909526d7ae9"
server_key_version: "encryptionkey@0"
I20260924 17:42:44.632346 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:44.633028 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:44.633214 21014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:44.633294 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0097684ec45b462d8471662e85a12b53"
format_stamp: "Formatted at 2026-09-24 17:42:44 on dist-test-slave-d0pf"
server_key: "72e502c541f1ad2a4cbfba20945c778d"
server_key_iv: "4faa3b7aa53f9c644e18f909526d7ae9"
server_key_version: "encryptionkey@0"
I20260924 17:42:44.633348 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:44.655748 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:44.658077 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:44.658200 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:44.662019 21014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34429
I20260924 17:42:44.662094 21081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34429 every 8 connection(s)
I20260924 17:42:44.662439 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:44.663337 21082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:44.665582 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53: Bootstrap starting.
I20260924 17:42:44.666164 21082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:44.666452 21082 log.cc:824] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:44.667169 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53: No bootstrap required, opened a new log
I20260924 17:42:44.668751 21082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260924 17:42:44.668915 21082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:44.668962 21082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0097684ec45b462d8471662e85a12b53, State: Initialized, Role: FOLLOWER
I20260924 17:42:44.669158 21082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260924 17:42:44.669209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21014
I20260924 17:42:44.669262 21082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:44.669304 21082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:44.669313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:44.669359 21082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:44.669471 392 external_mini_cluster.cc:1428] Setting key 58cf28ef6bdb87006695900abe765da7
I20260924 17:42:44.670178 21082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260924 17:42:44.670301 21082 leader_election.cc:302] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0097684ec45b462d8471662e85a12b53; no voters:
I20260924 17:42:44.670516 21082 leader_election.cc:288] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:44.670545 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:44.670737 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [term 1 LEADER]: Becoming Leader. State: Replica: 0097684ec45b462d8471662e85a12b53, State: Running, Role: LEADER
I20260924 17:42:44.670856 21082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:44.670877 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260924 17:42:44.671337 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0097684ec45b462d8471662e85a12b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } } }
I20260924 17:42:44.671479 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:44.672111 21096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:44.672616 21096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:44.672734 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0097684ec45b462d8471662e85a12b53. Latest consensus state: current_term: 1 leader_uuid: "0097684ec45b462d8471662e85a12b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0097684ec45b462d8471662e85a12b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } } }
I20260924 17:42:44.672855 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:44.674212 21096 catalog_manager.cc:1347] Generated new cluster ID: ffb77d032edc4edc943ea29c10f55929
I20260924 17:42:44.674264 21096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:44.676044 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:44.670386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40581 (local address 127.0.0.1:34429)
0924 17:42:44.670590 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:44.670600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:44.670635 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:42:44.670784 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:44.670791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:44.670846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:44.671009 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:44.671256 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:44.672704 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:44.673807 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:44.674403 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:44.674511 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:44.675248 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:44.675252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:44.675252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:44.675475 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:44.675515 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:44.675528 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:44.675595 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:44.675703 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 17:42:44.675853 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260924 17:42:44.676925 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:44.681739 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:44.678437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38487 (local address 127.0.0.1:34429)
0924 17:42:44.678471 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:44.678473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:44.678480 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:44.678501 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:44.678502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:44.678511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:44.678573 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:44.678713 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:44.679465 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:44.680441 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:44.680750 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:44.680821 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:44.681510 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:44.681511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:44.681512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:44.681516 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:44.681531 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:44.681536 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:44.681591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:44.681647 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 17:42:44.681671 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:42:44.692198 21096 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:44.692742 21096 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:44.694828 21096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0097684ec45b462d8471662e85a12b53: Generated new TSK 0
I20260924 17:42:44.695000 21096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:44.703107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21014
2026-09-24T17:42:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:44 dist-test-slave-d0pf krb5kdc[21113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:44 dist-test-slave-d0pf krb5kdc[21113](info): set up 2 sockets
Sep 24 17:42:44 dist-test-slave-d0pf krb5kdc[21113](info): commencing operation
krb5kdc: starting...
W20260924 17:42:45.743140 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:45 dist-test-slave-d0pf krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:45.861507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:45.939693 21126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:45.939889 21126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:45.939911 21126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:45.940963 21126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:45.940996 21126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:45.941011 21126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:45.941023 21126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:45.941035 21126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:45.942384 21126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:45.942559 21126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:45.942716 21126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:45.947306 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:45.947412 21126 server_base.cc:1034] running on GCE node
W20260924 17:42:45.947312 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:45.947414 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:45.947762 21126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:45.948032 21126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:45.949190 21126 hybrid_clock.cc:648] HybridClock initialized: now 1790271765949175 us; error 29 us; skew 500 ppm
Sep 24 17:42:45 dist-test-slave-d0pf krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:45.958781 21126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:45.959290 21126 webserver.cc:460] Webserver started at http://127.0.0.1:32843/ using document root <none> and password file <none>
I20260924 17:42:45.959512 21126 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:45.959583 21126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:45.959726 21126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:45.960749 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "ca52390096844b90812d35c2b41f1caa"
format_stamp: "Formatted at 2026-09-24 17:42:45 on dist-test-slave-d0pf"
server_key: "d2a692550307ab831d3e662830c75a1f"
server_key_iv: "abf96cc89670b9f0700b97a66eae5046"
server_key_version: "encryptionkey@0"
I20260924 17:42:45.961107 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca52390096844b90812d35c2b41f1caa"
format_stamp: "Formatted at 2026-09-24 17:42:45 on dist-test-slave-d0pf"
server_key: "d2a692550307ab831d3e662830c75a1f"
server_key_iv: "abf96cc89670b9f0700b97a66eae5046"
server_key_version: "encryptionkey@0"
I20260924 17:42:45.962412 21126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:45.963193 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:45.963357 21126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:45.963451 21126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "ca52390096844b90812d35c2b41f1caa"
format_stamp: "Formatted at 2026-09-24 17:42:45 on dist-test-slave-d0pf"
server_key: "d2a692550307ab831d3e662830c75a1f"
server_key_iv: "abf96cc89670b9f0700b97a66eae5046"
server_key_version: "encryptionkey@0"
I20260924 17:42:45.963529 21126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:45.966722 21126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:45.968940 21126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:45.969089 21126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:45.972813 21126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33961
I20260924 17:42:45.972841 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33961 every 8 connection(s)
I20260924 17:42:45.973142 21126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:45.974036 21194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:45.975950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21126
I20260924 17:42:45.976065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:45.976257 392 external_mini_cluster.cc:1428] Setting key f88cb87f292d81a937144c021aed7035
I20260924 17:42:45.976459 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: Bootstrap starting.
I20260924 17:42:45.977114 21194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:45.977388 21194 log.cc:824] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:45.978122 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: No bootstrap required, opened a new log
I20260924 17:42:45.978521 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:45.976744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37907 (local address 127.0.0.1:33961)
0924 17:42:45.976913 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:45.976922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:45.976953 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:42:45.977401 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:45.977408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:45.977447 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:45.977605 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:45.978156 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:45.978159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:45.978159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:45.978163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:45.978206 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:45.978219 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:45.978296 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:45.978334 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 17:42:45.978371 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260924 17:42:45.979384 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:45.980350 21194 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260924 17:42:45.980506 21194 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:45.980553 21194 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca52390096844b90812d35c2b41f1caa, State: Initialized, Role: FOLLOWER
I20260924 17:42:45.980667 21194 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260924 17:42:45.980756 21194 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:45.980824 21194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:45.980876 21194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:45.981518 21194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260924 17:42:45.981633 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:45.980668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52023 (local address 127.0.0.1:33961)
0924 17:42:45.980706 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:45.980709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:45.980793 (+ 84us) server_negotiation.cc:407] Connection header received
0924 17:42:45.980862 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:45.980862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:45.980867 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:45.980909 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:45.981439 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:45.981440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:45.981440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:45.981444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:45.981454 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:45.981458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:45.981503 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:45.981556 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 17:42:45.981569 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:42:45.981633 21194 leader_election.cc:302] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca52390096844b90812d35c2b41f1caa; no voters:
I20260924 17:42:45.981834 21194 leader_election.cc:288] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:45.981865 21204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:45.982025 21204 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [term 1 LEADER]: Becoming Leader. State: Replica: ca52390096844b90812d35c2b41f1caa, State: Running, Role: LEADER
I20260924 17:42:45.982056 21194 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:45.983276 21204 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260924 17:42:45.983777 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca52390096844b90812d35c2b41f1caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260924 17:42:45.983758 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca52390096844b90812d35c2b41f1caa. Latest consensus state: current_term: 1 leader_uuid: "ca52390096844b90812d35c2b41f1caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca52390096844b90812d35c2b41f1caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260924 17:42:45.983948 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:45.984153 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa [sys.catalog]: This master's current role is: LEADER
W20260924 17:42:45.984217 21218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:45.984272 21218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:45.984325 21220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:45.984501 21220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:45.985803 21220 catalog_manager.cc:1347] Generated new cluster ID: eb2b02d4891f40f3818462f329e98345
I20260924 17:42:45.985850 21220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:46.016072 21220 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:46.016265 21220 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:46.024591 21220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca52390096844b90812d35c2b41f1caa: Generated new TSK 0
I20260924 17:42:46.024814 21220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:46.027697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21126
2026-09-24T17:42:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:46 dist-test-slave-d0pf krb5kdc[21225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:46 dist-test-slave-d0pf krb5kdc[21225](info): set up 2 sockets
Sep 24 17:42:46 dist-test-slave-d0pf krb5kdc[21225](info): commencing operation
krb5kdc: starting...
W20260924 17:42:47.059190 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:47.178623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:47.259858 21238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:47.260126 21238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:47.260169 21238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:47.261353 21238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:47.261416 21238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:47.261442 21238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:47.261462 21238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:47.261493 21238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:47.262959 21238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:47.263242 21238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:47.263466 21238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:47.268265 21244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:47.268319 21238 server_base.cc:1034] running on GCE node
W20260924 17:42:47.268323 21243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:47.268368 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:47.268707 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:47.269035 21238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:47.270216 21238 hybrid_clock.cc:648] HybridClock initialized: now 1790271767270180 us; error 50 us; skew 500 ppm
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:47.272810 21238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:47.273309 21238 webserver.cc:460] Webserver started at http://127.0.0.1:34865/ using document root <none> and password file <none>
I20260924 17:42:47.273531 21238 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:47.273594 21238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:47.273737 21238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:47.274816 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "f18ecfeeed164309bf9c6349b4a6c7a7"
format_stamp: "Formatted at 2026-09-24 17:42:47 on dist-test-slave-d0pf"
server_key: "da59c602a327179995996ba009d0415d"
server_key_iv: "00e31ce87ec520c4c95f075a72af71bb"
server_key_version: "encryptionkey@0"
I20260924 17:42:47.275216 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "f18ecfeeed164309bf9c6349b4a6c7a7"
format_stamp: "Formatted at 2026-09-24 17:42:47 on dist-test-slave-d0pf"
server_key: "da59c602a327179995996ba009d0415d"
server_key_iv: "00e31ce87ec520c4c95f075a72af71bb"
server_key_version: "encryptionkey@0"
I20260924 17:42:47.276561 21238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:47.277349 21253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:47.277542 21238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:47.277640 21238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f18ecfeeed164309bf9c6349b4a6c7a7"
format_stamp: "Formatted at 2026-09-24 17:42:47 on dist-test-slave-d0pf"
server_key: "da59c602a327179995996ba009d0415d"
server_key_iv: "00e31ce87ec520c4c95f075a72af71bb"
server_key_version: "encryptionkey@0"
I20260924 17:42:47.277719 21238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:47.290702 21238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:47.293165 21238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:47.293344 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:47.297557 21238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32797
I20260924 17:42:47.297600 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32797 every 8 connection(s)
I20260924 17:42:47.298000 21238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:47.298888 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:47.301177 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7: Bootstrap starting.
I20260924 17:42:47.301720 21306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:47.301949 21306 log.cc:824] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:47.302632 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7: No bootstrap required, opened a new log
I20260924 17:42:47.303527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21238
I20260924 17:42:47.303625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:47.303810 392 external_mini_cluster.cc:1428] Setting key f073ec28890d3db3bfb3418a23fa6b77
I20260924 17:42:47.304744 21306 raft_consensus.cc:348] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } }
I20260924 17:42:47.304929 21306 raft_consensus.cc:374] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:47.304971 21306 raft_consensus.cc:729] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18ecfeeed164309bf9c6349b4a6c7a7, State: Initialized, Role: FOLLOWER
I20260924 17:42:47.305112 21306 consensus_queue.cc:260] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } }
I20260924 17:42:47.305198 21306 raft_consensus.cc:388] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:47.305253 21306 raft_consensus.cc:482] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:47.305303 21306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:47.306452 21306 raft_consensus.cc:504] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } }
I20260924 17:42:47.306569 21306 leader_election.cc:302] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18ecfeeed164309bf9c6349b4a6c7a7; no voters:
I20260924 17:42:47.306725 21306 leader_election.cc:288] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:47.306823 21311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:47.306972 21306 sys_catalog.cc:564] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:47.307013 21311 raft_consensus.cc:686] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [term 1 LEADER]: Becoming Leader. State: Replica: f18ecfeeed164309bf9c6349b4a6c7a7, State: Running, Role: LEADER
I20260924 17:42:47.307221 21311 consensus_queue.cc:237] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } }
I20260924 17:42:47.307654 21312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } } }
I20260924 17:42:47.307737 21312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:47.307967 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f18ecfeeed164309bf9c6349b4a6c7a7. Latest consensus state: current_term: 1 leader_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18ecfeeed164309bf9c6349b4a6c7a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32797 } } }
I20260924 17:42:47.308274 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:47.308463 21320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:47.309126 21320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:42:47.311834 21320 catalog_manager.cc:1347] Generated new cluster ID: 002ae287047c4b06b3107a3164330e4f
I20260924 17:42:47.311892 21320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:47.316938 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:47.304374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:49215 (local address 127.0.0.1:32797)
0924 17:42:47.304615 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:47.304625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:47.304668 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:42:47.304794 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:47.304801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:47.304862 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:47.305045 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:47.305812 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:47.307298 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:47.308488 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:47.309135 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:47.309279 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:47.312143 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:47.312152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:47.312152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:47.312163 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:47.314163 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:47.314693 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:47.314697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:47.314697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:47.315301 (+ 604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:47.315580 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:47.315581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:47.315581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:47.315938 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:47.316126 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:47.316589 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 17:42:47.316709 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":122,"threads_started":1}
I20260924 17:42:47.317898 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:47.329506 21320 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:47.330058 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:47.319273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:58147 (local address 127.0.0.1:32797)
0924 17:42:47.319315 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:47.319317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:47.323263 (+ 3946us) server_negotiation.cc:407] Connection header received
0924 17:42:47.323359 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:47.323360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:47.323372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:47.323422 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:47.324166 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:47.324691 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:47.325440 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:47.325672 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:47.325751 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:47.327184 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:47.327185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:47.327185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:47.327193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:47.328655 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:47.329092 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:47.329093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:47.329093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:47.329159 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:47.329442 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:47.329443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:47.329443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:47.329551 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:47.329662 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:47.329863 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 17:42:47.329941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:42:47.330395 21320 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:47.335698 21320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f18ecfeeed164309bf9c6349b4a6c7a7: Generated new TSK 0
I20260924 17:42:47.335953 21320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:47.341501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21238
W20260924 17:42:47.347402 21223 connection.cc:535] client connection to 127.0.0.1:32797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32797): Connection reset by peer (error 104)
2026-09-24T17:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21337](info): set up 2 sockets
Sep 24 17:42:47 dist-test-slave-d0pf krb5kdc[21337](info): commencing operation
krb5kdc: starting...
W20260924 17:42:48.379350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:48 dist-test-slave-d0pf krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:48.498558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:48.579664 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:48.579883 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:48.579907 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:48.581101 21350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:48.581149 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:48.581164 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:48.581178 21350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:48.581190 21350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:48.582674 21350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:48.582919 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:48.583173 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:48.588032 21355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:48.588027 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:48.588027 21356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:48.588486 21350 server_base.cc:1034] running on GCE node
I20260924 17:42:48.588743 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:48.589042 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:48.590206 21350 hybrid_clock.cc:648] HybridClock initialized: now 1790271768590208 us; error 42 us; skew 500 ppm
Sep 24 17:42:48 dist-test-slave-d0pf krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:48.592684 21350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:48.593179 21350 webserver.cc:460] Webserver started at http://127.0.0.1:36789/ using document root <none> and password file <none>
I20260924 17:42:48.593403 21350 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:48.593467 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:48.593617 21350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:48.594707 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "b00d8cdbe8cb44c08b29df9e2116203c"
format_stamp: "Formatted at 2026-09-24 17:42:48 on dist-test-slave-d0pf"
server_key: "ba5befa7ac57185082d82490b6043eef"
server_key_iv: "ff25f01a0e891b8ce0dcbad85b723dd5"
server_key_version: "encryptionkey@0"
I20260924 17:42:48.595145 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "b00d8cdbe8cb44c08b29df9e2116203c"
format_stamp: "Formatted at 2026-09-24 17:42:48 on dist-test-slave-d0pf"
server_key: "ba5befa7ac57185082d82490b6043eef"
server_key_iv: "ff25f01a0e891b8ce0dcbad85b723dd5"
server_key_version: "encryptionkey@0"
I20260924 17:42:48.596463 21350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:48.597167 21365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:48.597337 21350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:42:48.597429 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "b00d8cdbe8cb44c08b29df9e2116203c"
format_stamp: "Formatted at 2026-09-24 17:42:48 on dist-test-slave-d0pf"
server_key: "ba5befa7ac57185082d82490b6043eef"
server_key_iv: "ff25f01a0e891b8ce0dcbad85b723dd5"
server_key_version: "encryptionkey@0"
I20260924 17:42:48.597508 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:48.608837 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:48.611270 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:48.611428 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:48.615203 21350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41901
I20260924 17:42:48.615252 21417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41901 every 8 connection(s)
I20260924 17:42:48.615604 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:48.616468 21418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:48.618660 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c: Bootstrap starting.
I20260924 17:42:48.619259 21418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:48.619503 21418 log.cc:824] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:48.620141 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c: No bootstrap required, opened a new log
I20260924 17:42:48.621635 21418 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260924 17:42:48.621773 21418 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:48.621794 21418 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00d8cdbe8cb44c08b29df9e2116203c, State: Initialized, Role: FOLLOWER
I20260924 17:42:48.621892 21418 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260924 17:42:48.621974 21418 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:48.622012 21418 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:48.622066 21418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:48.622572 21418 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260924 17:42:48.622686 21418 leader_election.cc:302] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00d8cdbe8cb44c08b29df9e2116203c; no voters:
I20260924 17:42:48.622853 21418 leader_election.cc:288] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:48.622892 21421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:48.623093 21421 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [term 1 LEADER]: Becoming Leader. State: Replica: b00d8cdbe8cb44c08b29df9e2116203c, State: Running, Role: LEADER
I20260924 17:42:48.623121 21418 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:48.623193 21421 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260924 17:42:48.623358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21350
I20260924 17:42:48.623471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:48.623582 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } } }
I20260924 17:42:48.623670 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:48.623687 392 external_mini_cluster.cc:1428] Setting key 9071c58d867d327aa8f20eba9c2e14c5
I20260924 17:42:48.623867 21425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:48.624418 21425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:48.624641 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00d8cdbe8cb44c08b29df9e2116203c. Latest consensus state: current_term: 1 leader_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d8cdbe8cb44c08b29df9e2116203c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } } }
I20260924 17:42:48.624835 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:48.626122 21425 catalog_manager.cc:1347] Generated new cluster ID: e4a80bf7114a40a1a50c1f25ccb0493e
I20260924 17:42:48.626173 21425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:48.630033 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:48.624327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:50987 (local address 127.0.0.1:41901)
0924 17:42:48.624713 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:48.624722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:48.624755 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:42:48.624860 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:48.624867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:48.624938 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:48.625065 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:48.625387 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:48.626800 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:48.627938 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:48.628535 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:48.628638 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:48.629360 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:48.629364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:48.629364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:48.629558 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:48.629603 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:48.629616 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:48.629690 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:48.629813 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 17:42:48.629839 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":55,"threads_started":1}
I20260924 17:42:48.630903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:48.635823 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:48.632293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:60019 (local address 127.0.0.1:41901)
0924 17:42:48.632325 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:48.632326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:48.632439 (+ 113us) server_negotiation.cc:407] Connection header received
0924 17:42:48.632515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:48.632516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:48.632526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:48.632552 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:48.632736 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:48.633435 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:48.634405 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:48.634732 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:48.634766 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:48.635556 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:48.635557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:48.635558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:48.635562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:48.635587 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:48.635591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:48.635621 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:48.635735 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 17:42:48.635748 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 17:42:48.666749 21425 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:48.667384 21425 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:48.670009 21425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00d8cdbe8cb44c08b29df9e2116203c: Generated new TSK 0
I20260924 17:42:48.670185 21425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:48.680402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21350
W20260924 17:42:48.684808 21335 connection.cc:535] client connection to 127.0.0.1:41901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41901): Connection reset by peer (error 104)
2026-09-24T17:42:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:48 dist-test-slave-d0pf krb5kdc[21449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:48 dist-test-slave-d0pf krb5kdc[21449](info): set up 2 sockets
Sep 24 17:42:48 dist-test-slave-d0pf krb5kdc[21449](info): commencing operation
krb5kdc: starting...
W20260924 17:42:49.711086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:49 dist-test-slave-d0pf krb5kdc[21449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:49.830266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:49.911520 21462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:49.911744 21462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:49.911770 21462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:49.913352 21462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:49.913416 21462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:49.913440 21462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:49.913465 21462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:49.913494 21462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:49.915067 21462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:49.915330 21462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:49.915578 21462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:49.920694 21468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:49.920761 21467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:49.920776 21462 server_base.cc:1034] running on GCE node
W20260924 17:42:49.920882 21470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:49.921123 21462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:49.921460 21462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:49.922623 21462 hybrid_clock.cc:648] HybridClock initialized: now 1790271769922606 us; error 46 us; skew 500 ppm
Sep 24 17:42:49 dist-test-slave-d0pf krb5kdc[21449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:49.925287 21462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:49.925706 21462 webserver.cc:460] Webserver started at http://127.0.0.1:34705/ using document root <none> and password file <none>
I20260924 17:42:49.925940 21462 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:49.925987 21462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:49.926144 21462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:49.927294 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "85b1603cb34b460c80e3de4ac739bde9"
format_stamp: "Formatted at 2026-09-24 17:42:49 on dist-test-slave-d0pf"
server_key: "804bfb9d97176a650468be05fcc5a71b"
server_key_iv: "2ece482ec951f71494e863b5a5b047ac"
server_key_version: "encryptionkey@0"
I20260924 17:42:49.927652 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "85b1603cb34b460c80e3de4ac739bde9"
format_stamp: "Formatted at 2026-09-24 17:42:49 on dist-test-slave-d0pf"
server_key: "804bfb9d97176a650468be05fcc5a71b"
server_key_iv: "2ece482ec951f71494e863b5a5b047ac"
server_key_version: "encryptionkey@0"
I20260924 17:42:49.929021 21462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:49.929913 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:49.930127 21462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:49.930233 21462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "85b1603cb34b460c80e3de4ac739bde9"
format_stamp: "Formatted at 2026-09-24 17:42:49 on dist-test-slave-d0pf"
server_key: "804bfb9d97176a650468be05fcc5a71b"
server_key_iv: "2ece482ec951f71494e863b5a5b047ac"
server_key_version: "encryptionkey@0"
I20260924 17:42:49.930295 21462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:49.933938 21462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:49.936128 21462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:49.936249 21462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:49.940317 21462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32941
I20260924 17:42:49.940349 21529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32941 every 8 connection(s)
I20260924 17:42:49.940711 21462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:49.941748 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:49.944054 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: Bootstrap starting.
I20260924 17:42:49.944620 21530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:49.944895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21462
I20260924 17:42:49.944918 21530 log.cc:824] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:49.945036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:49.945237 392 external_mini_cluster.cc:1428] Setting key aa61d1b7bd3d404f2e42942fd6ef8d31
I20260924 17:42:49.945576 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: No bootstrap required, opened a new log
I20260924 17:42:49.947937 21530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } }
I20260924 17:42:49.948133 21530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:49.948202 21530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b1603cb34b460c80e3de4ac739bde9, State: Initialized, Role: FOLLOWER
I20260924 17:42:49.948333 21530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } }
I20260924 17:42:49.948437 21530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:49.948482 21530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:49.948530 21530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:49.948585 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:49.945748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:49871 (local address 127.0.0.1:32941)
0924 17:42:49.945940 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:49.945949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:49.945976 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:42:49.946964 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:49.946972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:49.947001 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:49.947201 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:49.947849 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:49.947852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:49.947852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:49.947856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:49.947897 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:49.947908 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:49.947966 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:49.948060 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 17:42:49.948126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260924 17:42:49.949458 21530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } }
I20260924 17:42:49.949553 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:49.949584 21530 leader_election.cc:302] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b1603cb34b460c80e3de4ac739bde9; no voters:
I20260924 17:42:49.949748 21530 leader_election.cc:288] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:49.949811 21535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:49.949968 21530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:49.949993 21535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [term 1 LEADER]: Becoming Leader. State: Replica: 85b1603cb34b460c80e3de4ac739bde9, State: Running, Role: LEADER
I20260924 17:42:49.950088 21535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } }
I20260924 17:42:49.950469 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b1603cb34b460c80e3de4ac739bde9. Latest consensus state: current_term: 1 leader_uuid: "85b1603cb34b460c80e3de4ac739bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } } }
I20260924 17:42:49.950727 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:49.950639 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b1603cb34b460c80e3de4ac739bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1603cb34b460c80e3de4ac739bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32941 } } }
I20260924 17:42:49.951176 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9 [sys.catalog]: This master's current role is: LEADER
W20260924 17:42:49.951843 21555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:49.951906 21555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:49.951979 21548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:49.952173 21548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:49.952728 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:49.951221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:44255 (local address 127.0.0.1:32941)
0924 17:42:49.951265 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:49.951267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:49.951509 (+ 242us) server_negotiation.cc:407] Connection header received
0924 17:42:49.951655 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:49.951656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:49.951661 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:49.951733 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:49.952516 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:49.952516 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:49.952517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:49.952520 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:49.952532 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:49.952536 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:49.952580 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:49.952641 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:42:49.952652 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:42:49.953840 21548 catalog_manager.cc:1347] Generated new cluster ID: a1b401be6d144d59b14bce98cf052bed
I20260924 17:42:49.953894 21548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:49.966321 21548 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:49.966506 21548 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:49.969066 21548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b1603cb34b460c80e3de4ac739bde9: Generated new TSK 0
I20260924 17:42:49.969259 21548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:49.977965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21462
2026-09-24T17:42:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:50 dist-test-slave-d0pf krb5kdc[21561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:50 dist-test-slave-d0pf krb5kdc[21561](info): set up 2 sockets
Sep 24 17:42:50 dist-test-slave-d0pf krb5kdc[21561](info): commencing operation
krb5kdc: starting...
W20260924 17:42:51.019157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:51.137889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:51.219547 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:51.219766 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:51.219790 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:51.220929 21574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:51.220969 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:51.220984 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:51.220996 21574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:51.221009 21574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:51.222402 21574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:51.222594 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:51.222765 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:51.227626 21579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:51.227579 21580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:51.227607 21574 server_base.cc:1034] running on GCE node
W20260924 17:42:51.227576 21582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:51.228039 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:51.228355 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:51.229514 21574 hybrid_clock.cc:648] HybridClock initialized: now 1790271771229499 us; error 46 us; skew 500 ppm
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:51.238448 21574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:51.239004 21574 webserver.cc:460] Webserver started at http://127.0.0.1:35129/ using document root <none> and password file <none>
I20260924 17:42:51.239287 21574 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:51.239354 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:51.239506 21574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:51.240605 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "7a23e59653354659be7dcfbf08727afe"
format_stamp: "Formatted at 2026-09-24 17:42:51 on dist-test-slave-d0pf"
server_key: "7b94a8dc5aec9bc414268e14dd4eedde"
server_key_iv: "7f0311fde320c25e99d5fe270d4945fe"
server_key_version: "encryptionkey@0"
I20260924 17:42:51.240983 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a23e59653354659be7dcfbf08727afe"
format_stamp: "Formatted at 2026-09-24 17:42:51 on dist-test-slave-d0pf"
server_key: "7b94a8dc5aec9bc414268e14dd4eedde"
server_key_iv: "7f0311fde320c25e99d5fe270d4945fe"
server_key_version: "encryptionkey@0"
I20260924 17:42:51.242445 21574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:42:51.243257 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:51.243427 21574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:51.243536 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "7a23e59653354659be7dcfbf08727afe"
format_stamp: "Formatted at 2026-09-24 17:42:51 on dist-test-slave-d0pf"
server_key: "7b94a8dc5aec9bc414268e14dd4eedde"
server_key_iv: "7f0311fde320c25e99d5fe270d4945fe"
server_key_version: "encryptionkey@0"
I20260924 17:42:51.243613 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:51.255434 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:51.257525 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:51.257699 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:51.261713 21574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43689
I20260924 17:42:51.261749 21641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43689 every 8 connection(s)
I20260924 17:42:51.262126 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:51.262620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21574
I20260924 17:42:51.262763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:51.262980 392 external_mini_cluster.cc:1428] Setting key 51be82f670c6b1ee3e0ca43ef764c7f4
I20260924 17:42:51.263247 21642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:51.266583 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe: Bootstrap starting.
I20260924 17:42:51.267293 21642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:51.267606 21642 log.cc:824] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:51.268276 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe: No bootstrap required, opened a new log
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:42:51.270123 21642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } }
I20260924 17:42:51.270277 21642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:51.270320 21642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a23e59653354659be7dcfbf08727afe, State: Initialized, Role: FOLLOWER
I20260924 17:42:51.270471 21642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } }
I20260924 17:42:51.270550 21642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:51.270591 21642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:51.270638 21642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:51.271220 21642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } }
I20260924 17:42:51.271335 21642 leader_election.cc:302] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a23e59653354659be7dcfbf08727afe; no voters:
I20260924 17:42:51.271528 21642 leader_election.cc:288] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:51.271562 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:51.271667 21647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [term 1 LEADER]: Becoming Leader. State: Replica: 7a23e59653354659be7dcfbf08727afe, State: Running, Role: LEADER
I20260924 17:42:51.271800 21642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:51.271800 21647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } }
I20260924 17:42:51.272334 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a23e59653354659be7dcfbf08727afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } } }
I20260924 17:42:51.272332 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a23e59653354659be7dcfbf08727afe. Latest consensus state: current_term: 1 leader_uuid: "7a23e59653354659be7dcfbf08727afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a23e59653354659be7dcfbf08727afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43689 } } }
I20260924 17:42:51.272444 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:51.272465 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:51.272677 21655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:51.273182 21655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:51.274503 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:51.263519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:59359 (local address 127.0.0.1:43689)
0924 17:42:51.263690 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:51.263700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:51.263733 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:42:51.264422 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:51.264429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:51.264480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:51.264606 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:51.265459 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:51.266361 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:51.267441 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:51.267881 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:51.268000 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:51.270297 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:51.270301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:51.270301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:51.270321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:51.272459 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:51.273061 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:51.273064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:51.273065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:51.273170 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:51.273461 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:51.273462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:51.273462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:51.273773 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:51.273909 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:51.274198 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 17:42:51.274313 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 17:42:51.275009 21664 mvcc.cc:205] Tried to move back new op lower bound from 7332953175138807808 to 7332953175129980928. Current Snapshot: MvccSnapshot[applied={T|T < 7332953175138807808}]
I20260924 17:42:51.275573 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:51.275667 21655 catalog_manager.cc:1347] Generated new cluster ID: 8735439a3e634d7b87dce4700f336345
I20260924 17:42:51.275699 21655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:51.286393 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:51.277003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:56901 (local address 127.0.0.1:43689)
0924 17:42:51.279101 (+ 2098us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:51.279103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:51.279110 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:42:51.279130 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:51.279130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:51.279140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:51.279187 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:51.279852 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:51.280630 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:51.281610 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:51.281955 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:51.282034 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:51.283464 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:51.283465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:51.283465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:51.283472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:51.285080 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:51.285477 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:51.285478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:51.285478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:51.285537 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:51.285753 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:51.285754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:51.285754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:51.285888 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:51.285990 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:51.286222 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 17:42:51.286290 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078}
I20260924 17:42:51.290249 21655 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:51.290784 21655 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:51.299979 21655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a23e59653354659be7dcfbf08727afe: Generated new TSK 0
I20260924 17:42:51.300180 21655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:51.306057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21574
W20260924 17:42:51.310328 21559 connection.cc:535] client connection to 127.0.0.1:43689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43689): Connection reset by peer (error 104)
2026-09-24T17:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21673](info): set up 2 sockets
Sep 24 17:42:51 dist-test-slave-d0pf krb5kdc[21673](info): commencing operation
krb5kdc: starting...
W20260924 17:42:52.335866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:52 dist-test-slave-d0pf krb5kdc[21673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:52.454360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:52.534658 21686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:52.534878 21686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:52.534904 21686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:52.536111 21686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:52.536157 21686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:52.536171 21686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:52.536223 21686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:52.536244 21686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:52.538210 21686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:52.538468 21686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:52.538699 21686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:52.543457 21694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:52.543450 21692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:52.543604 21686 server_base.cc:1034] running on GCE node
W20260924 17:42:52.543450 21691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:52.543929 21686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:52.544179 21686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:52.545323 21686 hybrid_clock.cc:648] HybridClock initialized: now 1790271772545310 us; error 31 us; skew 500 ppm
Sep 24 17:42:52 dist-test-slave-d0pf krb5kdc[21673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:52.547984 21686 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:52.548475 21686 webserver.cc:460] Webserver started at http://127.0.0.1:35793/ using document root <none> and password file <none>
I20260924 17:42:52.548707 21686 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:52.548753 21686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:52.548923 21686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:52.550002 21686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "d354d00d4b44442a9029ef55091f06d4"
format_stamp: "Formatted at 2026-09-24 17:42:52 on dist-test-slave-d0pf"
server_key: "6a5efb3d30bb701fff7e3d376fe417c4"
server_key_iv: "9e6aabcac79d634a18d5450f49726b76"
server_key_version: "encryptionkey@0"
I20260924 17:42:52.550359 21686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "d354d00d4b44442a9029ef55091f06d4"
format_stamp: "Formatted at 2026-09-24 17:42:52 on dist-test-slave-d0pf"
server_key: "6a5efb3d30bb701fff7e3d376fe417c4"
server_key_iv: "9e6aabcac79d634a18d5450f49726b76"
server_key_version: "encryptionkey@0"
I20260924 17:42:52.551795 21686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:52.552742 21701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:52.552955 21686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:52.553068 21686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "d354d00d4b44442a9029ef55091f06d4"
format_stamp: "Formatted at 2026-09-24 17:42:52 on dist-test-slave-d0pf"
server_key: "6a5efb3d30bb701fff7e3d376fe417c4"
server_key_iv: "9e6aabcac79d634a18d5450f49726b76"
server_key_version: "encryptionkey@0"
I20260924 17:42:52.553146 21686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:52.564915 21686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:52.567276 21686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:52.567446 21686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:52.571447 21686 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35151
I20260924 17:42:52.571484 21753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35151 every 8 connection(s)
I20260924 17:42:52.571834 21686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:52.572784 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:52.575006 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4: Bootstrap starting.
I20260924 17:42:52.575598 21754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:52.575877 21754 log.cc:824] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:52.576583 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4: No bootstrap required, opened a new log
I20260924 17:42:52.578109 21754 raft_consensus.cc:348] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 17:42:52.578248 21754 raft_consensus.cc:374] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:52.578269 21754 raft_consensus.cc:729] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d354d00d4b44442a9029ef55091f06d4, State: Initialized, Role: FOLLOWER
I20260924 17:42:52.578433 21754 consensus_queue.cc:260] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 17:42:52.578501 21754 raft_consensus.cc:388] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:52.578523 21754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:52.578586 21754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:52.579192 21754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 17:42:52.579313 21754 leader_election.cc:302] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d354d00d4b44442a9029ef55091f06d4; no voters:
I20260924 17:42:52.579380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21686
I20260924 17:42:52.579497 21754 leader_election.cc:288] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:52.579504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:52.579661 21758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:52.579710 392 external_mini_cluster.cc:1428] Setting key 4074d1171a915a35d554171d45ce3dee
I20260924 17:42:52.579756 21754 sys_catalog.cc:564] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:52.579816 21758 raft_consensus.cc:686] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [term 1 LEADER]: Becoming Leader. State: Replica: d354d00d4b44442a9029ef55091f06d4, State: Running, Role: LEADER
I20260924 17:42:52.579943 21758 consensus_queue.cc:237] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 17:42:52.580250 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d354d00d4b44442a9029ef55091f06d4. Latest consensus state: current_term: 1 leader_uuid: "d354d00d4b44442a9029ef55091f06d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } } }
I20260924 17:42:52.580379 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:52.580410 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d354d00d4b44442a9029ef55091f06d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d354d00d4b44442a9029ef55091f06d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } } }
I20260924 17:42:52.580539 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:52.580646 21768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:52.581236 21768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:52.583083 21768 catalog_manager.cc:1347] Generated new cluster ID: 99bc59a13aa443ccb24c59550da0c06c
I20260924 17:42:52.583137 21768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:52.586616 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:52.580360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:47903 (local address 127.0.0.1:35151)
0924 17:42:52.581171 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:52.581180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:52.581214 (+ 34us) server_negotiation.cc:407] Connection header received
0924 17:42:52.581345 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:52.581353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:52.581500 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:52.581684 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:52.581945 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:52.583327 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:52.584433 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:52.585077 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:52.585189 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:52.585915 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:52.585919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:52.585919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:52.586138 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:52.586178 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:52.586195 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:52.586289 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:52.586365 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 17:42:52.586392 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":80,"threads_started":1}
I20260924 17:42:52.587594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:52.589792 21768 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:52.590468 21768 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:52.592698 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:52.589276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:39597 (local address 127.0.0.1:35151)
0924 17:42:52.589307 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:52.589309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:52.589317 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:42:52.589376 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:52.589377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:52.589388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:52.589451 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:52.589582 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:52.590309 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:52.591372 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:52.591678 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:52.591746 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:52.592447 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:52.592448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:52.592448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:52.592453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:52.592473 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:52.592478 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:52.592533 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:52.592612 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 17:42:52.592625 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:42:52.598636 21768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d354d00d4b44442a9029ef55091f06d4: Generated new TSK 0
I20260924 17:42:52.598820 21768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:52.601640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21686
W20260924 17:42:52.605854 21671 connection.cc:535] client connection to 127.0.0.1:35151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35151): Connection reset by peer (error 104)
2026-09-24T17:42:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:52 dist-test-slave-d0pf krb5kdc[21785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:52 dist-test-slave-d0pf krb5kdc[21785](info): set up 2 sockets
Sep 24 17:42:52 dist-test-slave-d0pf krb5kdc[21785](info): commencing operation
krb5kdc: starting...
W20260924 17:42:53.651197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:53 dist-test-slave-d0pf krb5kdc[21785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:42:53.777812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:53.858480 21798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:53.858701 21798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:53.858726 21798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:53.859958 21798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:53.860008 21798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:53.860021 21798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:53.860035 21798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:53.860049 21798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:53.861600 21798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:53.861879 21798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:53.862147 21798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:53.866853 21803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:53.866849 21806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:53.866871 21804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:53.866930 21798 server_base.cc:1034] running on GCE node
I20260924 17:42:53.867362 21798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:53.867666 21798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:53.868810 21798 hybrid_clock.cc:648] HybridClock initialized: now 1790271773868793 us; error 37 us; skew 500 ppm
Sep 24 17:42:53 dist-test-slave-d0pf krb5kdc[21785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:53.871447 21798 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:42:53.871919 21798 webserver.cc:460] Webserver started at http://127.0.0.1:41701/ using document root <none> and password file <none>
I20260924 17:42:53.872155 21798 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:53.872200 21798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:53.872339 21798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:53.873499 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "5277cf96784d4fba96a869754f969f4b"
format_stamp: "Formatted at 2026-09-24 17:42:53 on dist-test-slave-d0pf"
server_key: "4399d2c6de7cc7ade4a4b4e441c57425"
server_key_iv: "ab9c733c7212b3e0432382800101e4c0"
server_key_version: "encryptionkey@0"
I20260924 17:42:53.873854 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "5277cf96784d4fba96a869754f969f4b"
format_stamp: "Formatted at 2026-09-24 17:42:53 on dist-test-slave-d0pf"
server_key: "4399d2c6de7cc7ade4a4b4e441c57425"
server_key_iv: "ab9c733c7212b3e0432382800101e4c0"
server_key_version: "encryptionkey@0"
I20260924 17:42:53.875200 21798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:42:53.875898 21813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:53.876075 21798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:53.876164 21798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "5277cf96784d4fba96a869754f969f4b"
format_stamp: "Formatted at 2026-09-24 17:42:53 on dist-test-slave-d0pf"
server_key: "4399d2c6de7cc7ade4a4b4e441c57425"
server_key_iv: "ab9c733c7212b3e0432382800101e4c0"
server_key_version: "encryptionkey@0"
I20260924 17:42:53.876235 21798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:53.879500 21798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:53.881784 21798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:53.881948 21798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:53.886014 21798 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35881
I20260924 17:42:53.886044 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35881 every 8 connection(s)
I20260924 17:42:53.886387 21798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:53.887436 21866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:53.889676 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b: Bootstrap starting.
I20260924 17:42:53.890250 21866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:53.890537 21866 log.cc:824] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:53.891294 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b: No bootstrap required, opened a new log
I20260924 17:42:53.892457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21798
I20260924 17:42:53.892573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:53.892747 392 external_mini_cluster.cc:1428] Setting key 69b3f8ecf456ed87ce8e9ece6bef5e0f
I20260924 17:42:53.893359 21866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } }
I20260924 17:42:53.893558 21866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:53.893607 21866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5277cf96784d4fba96a869754f969f4b, State: Initialized, Role: FOLLOWER
I20260924 17:42:53.893738 21866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } }
I20260924 17:42:53.893996 21866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:53.894097 21866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:53.894160 21866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:53.894997 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:53.893305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:58535 (local address 127.0.0.1:35881)
0924 17:42:53.893480 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:53.893489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:53.893525 (+ 36us) server_negotiation.cc:407] Connection header received
0924 17:42:53.893648 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:53.893656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:53.893687 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:53.893826 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:53.894534 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:53.894537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:53.894537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:53.894541 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:53.894577 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:53.894588 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:53.894643 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:53.894735 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 17:42:53.894769 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 17:42:53.895361 21866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } }
I20260924 17:42:53.895473 21866 leader_election.cc:302] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5277cf96784d4fba96a869754f969f4b; no voters:
I20260924 17:42:53.895627 21866 leader_election.cc:288] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:53.895653 21871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:53.895849 21871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [term 1 LEADER]: Becoming Leader. State: Replica: 5277cf96784d4fba96a869754f969f4b, State: Running, Role: LEADER
I20260924 17:42:53.895888 21866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:53.895969 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:53.896014 21871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } }
I20260924 17:42:53.896323 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5277cf96784d4fba96a869754f969f4b. Latest consensus state: current_term: 1 leader_uuid: "5277cf96784d4fba96a869754f969f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } } }
I20260924 17:42:53.896394 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:53.896543 21872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5277cf96784d4fba96a869754f969f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277cf96784d4fba96a869754f969f4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35881 } } }
I20260924 17:42:53.896600 21872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:53.896718 21877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:53.897225 21877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:53.899020 21877 catalog_manager.cc:1347] Generated new cluster ID: b6ae7d6a26d24e5480954c010dfb1750
I20260924 17:42:53.899096 21877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:53.899237 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:53.897501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:34741 (local address 127.0.0.1:35881)
0924 17:42:53.897537 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:53.897539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:53.897802 (+ 263us) server_negotiation.cc:407] Connection header received
0924 17:42:53.897889 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:53.897890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:53.897919 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:53.897982 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:53.898981 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:53.898982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:53.898982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:53.898987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:53.899002 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:53.899006 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:53.899094 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:53.899167 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 17:42:53.899182 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:42:53.918248 21877 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:53.918488 21877 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:53.926342 21877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5277cf96784d4fba96a869754f969f4b: Generated new TSK 0
I20260924 17:42:53.926579 21877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:53.928586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21798
2026-09-24T17:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:53 dist-test-slave-d0pf krb5kdc[21897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:53 dist-test-slave-d0pf krb5kdc[21897](info): set up 2 sockets
Sep 24 17:42:53 dist-test-slave-d0pf krb5kdc[21897](info): commencing operation
krb5kdc: starting...
W20260924 17:42:54.963250 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:55.083887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37005
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:55.169418 21910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:55.169639 21910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:55.169664 21910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:55.170872 21910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:55.170915 21910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:55.170930 21910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:55.170970 21910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:55.170997 21910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:55.172529 21910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:55.172801 21910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:55.173038 21910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:55.178270 21916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:55.178270 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:55.178447 21910 server_base.cc:1034] running on GCE node
W20260924 17:42:55.178320 21918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:55.178797 21910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:55.179159 21910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:55.180341 21910 hybrid_clock.cc:648] HybridClock initialized: now 1790271775180326 us; error 42 us; skew 500 ppm
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:55.188611 21910 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:55.189230 21910 webserver.cc:460] Webserver started at http://127.0.98.62:43499/ using document root <none> and password file <none>
I20260924 17:42:55.189532 21910 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:55.189607 21910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:55.189757 21910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:55.190851 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "bdf75cea86f4422a82ec9ca6b2558008"
format_stamp: "Formatted at 2026-09-24 17:42:55 on dist-test-slave-d0pf"
server_key: "28ec250099c68842287106c42218dc64"
server_key_iv: "39cb61f6847653a71e3ee2a39892bb6c"
server_key_version: "encryptionkey@0"
I20260924 17:42:55.191236 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdf75cea86f4422a82ec9ca6b2558008"
format_stamp: "Formatted at 2026-09-24 17:42:55 on dist-test-slave-d0pf"
server_key: "28ec250099c68842287106c42218dc64"
server_key_iv: "39cb61f6847653a71e3ee2a39892bb6c"
server_key_version: "encryptionkey@0"
I20260924 17:42:55.192597 21910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:55.193468 21925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:55.193660 21910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:42:55.193769 21910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "bdf75cea86f4422a82ec9ca6b2558008"
format_stamp: "Formatted at 2026-09-24 17:42:55 on dist-test-slave-d0pf"
server_key: "28ec250099c68842287106c42218dc64"
server_key_iv: "39cb61f6847653a71e3ee2a39892bb6c"
server_key_version: "encryptionkey@0"
I20260924 17:42:55.193845 21910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:55.212937 21910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:55.214884 21910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:55.215072 21910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:55.219316 21910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37005
I20260924 17:42:55.219354 21977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37005 every 8 connection(s)
I20260924 17:42:55.219766 21910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:55.220606 21978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:55.222751 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008: Bootstrap starting.
I20260924 17:42:55.223357 21978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:55.223588 21978 log.cc:824] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:55.224213 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008: No bootstrap required, opened a new log
I20260924 17:42:55.225687 21978 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } }
I20260924 17:42:55.225818 21978 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:55.225840 21978 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf75cea86f4422a82ec9ca6b2558008, State: Initialized, Role: FOLLOWER
I20260924 17:42:55.225947 21978 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } }
I20260924 17:42:55.226001 21978 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:55.226020 21978 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:55.226078 21978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:55.226578 21978 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } }
I20260924 17:42:55.226662 21978 leader_election.cc:302] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf75cea86f4422a82ec9ca6b2558008; no voters:
I20260924 17:42:55.226837 21978 leader_election.cc:288] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:55.226873 21981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:55.227097 21981 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [term 1 LEADER]: Becoming Leader. State: Replica: bdf75cea86f4422a82ec9ca6b2558008, State: Running, Role: LEADER
I20260924 17:42:55.227137 21978 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:55.227217 21981 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } }
I20260924 17:42:55.227648 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf75cea86f4422a82ec9ca6b2558008. Latest consensus state: current_term: 1 leader_uuid: "bdf75cea86f4422a82ec9ca6b2558008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } } }
I20260924 17:42:55.227730 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:55.227783 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf75cea86f4422a82ec9ca6b2558008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf75cea86f4422a82ec9ca6b2558008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37005 } } }
I20260924 17:42:55.227854 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:55.228193 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:55.228711 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:55.229061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 21910
I20260924 17:42:55.229183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:55.229359 392 external_mini_cluster.cc:1428] Setting key 02c60f2ab3eca268025b2cee0832f64e
I20260924 17:42:55.230340 21990 catalog_manager.cc:1347] Generated new cluster ID: a7b10dd11e344a3e8c7093a8c4a74d41
I20260924 17:42:55.230410 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:55.239281 21990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:55.239754 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:55.241672 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:55.229801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56375 (local address 127.0.98.62:37005)
0924 17:42:55.229951 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:55.229960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:55.230525 (+ 565us) server_negotiation.cc:407] Connection header received
0924 17:42:55.230666 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:55.230673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:55.230724 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:55.230869 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:55.231713 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:55.233115 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:55.234212 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:55.234810 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:55.234902 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:55.237794 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:55.237798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:55.237799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:55.237810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:55.239882 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:55.240333 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:55.240336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:55.240337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:55.240447 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:55.240676 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:55.240676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:55.240677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:55.240948 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:55.241119 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:55.241201 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:42:55.241438 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 17:42:55.242491 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:55.252091 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:55.243950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53701 (local address 127.0.98.62:37005)
0924 17:42:55.243983 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:55.243985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:55.244095 (+ 110us) server_negotiation.cc:407] Connection header received
0924 17:42:55.244184 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:55.244184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:55.244193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:55.244221 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:55.244942 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:55.245767 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:55.246926 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:55.247278 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:55.247321 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:55.249089 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:55.249091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:55.249091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:55.249098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:55.250697 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:55.251188 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:55.251189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:55.251189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:55.251246 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:55.251489 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:55.251489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:55.251490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:55.251615 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:55.251694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:55.251930 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 17:42:55.251996 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 17:42:55.252851 21990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf75cea86f4422a82ec9ca6b2558008: Generated new TSK 0
I20260924 17:42:55.253063 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:55.258652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 21910
W20260924 17:42:55.262398 21895 connection.cc:535] client connection to 127.0.98.62:37005 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37005): Connection reset by peer (error 104)
2026-09-24T17:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[22009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[22009](info): set up 2 sockets
Sep 24 17:42:55 dist-test-slave-d0pf krb5kdc[22009](info): commencing operation
krb5kdc: starting...
W20260924 17:42:56.311220 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:56 dist-test-slave-d0pf krb5kdc[22009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:56.430702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:56.511749 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:56.512002 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:56.512039 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:56.513258 22022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:56.513331 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:56.513355 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:56.513386 22022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:56.513429 22022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:56.514971 22022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:56.515282 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:56.515512 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:56.520304 22030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:56.520380 22027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:56.520514 22028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:56.520396 22022 server_base.cc:1034] running on GCE node
I20260924 17:42:56.520767 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:56.521039 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:56.522228 22022 hybrid_clock.cc:648] HybridClock initialized: now 1790271776522214 us; error 40 us; skew 500 ppm
Sep 24 17:42:56 dist-test-slave-d0pf krb5kdc[22009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:56.529435 22022 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:56.529953 22022 webserver.cc:460] Webserver started at http://127.0.98.62:41747/ using document root <none> and password file <none>
I20260924 17:42:56.530190 22022 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:56.530265 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:56.530422 22022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:56.531598 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "29e6b8ca56b14da382741457910c070a"
format_stamp: "Formatted at 2026-09-24 17:42:56 on dist-test-slave-d0pf"
server_key: "63f968f05006252aa434c0e458b97c42"
server_key_iv: "71d4f4f5060526df42eac508225ab20f"
server_key_version: "encryptionkey@0"
I20260924 17:42:56.531968 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "29e6b8ca56b14da382741457910c070a"
format_stamp: "Formatted at 2026-09-24 17:42:56 on dist-test-slave-d0pf"
server_key: "63f968f05006252aa434c0e458b97c42"
server_key_iv: "71d4f4f5060526df42eac508225ab20f"
server_key_version: "encryptionkey@0"
I20260924 17:42:56.533304 22022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:42:56.534070 22037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:56.534257 22022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:56.534368 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "29e6b8ca56b14da382741457910c070a"
format_stamp: "Formatted at 2026-09-24 17:42:56 on dist-test-slave-d0pf"
server_key: "63f968f05006252aa434c0e458b97c42"
server_key_iv: "71d4f4f5060526df42eac508225ab20f"
server_key_version: "encryptionkey@0"
I20260924 17:42:56.534443 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:56.543193 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:56.544986 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:56.545148 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:56.549093 22022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43601
I20260924 17:42:56.549127 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43601 every 8 connection(s)
I20260924 17:42:56.549495 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:56.550364 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:56.552616 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: Bootstrap starting.
I20260924 17:42:56.553149 22090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:56.553375 22090 log.cc:824] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:56.554071 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: No bootstrap required, opened a new log
I20260924 17:42:56.555593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22022
I20260924 17:42:56.555725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:56.555925 392 external_mini_cluster.cc:1428] Setting key 49d342da7a2c0f008e1eeace72935668
I20260924 17:42:56.556021 22090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260924 17:42:56.556196 22090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:56.556221 22090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e6b8ca56b14da382741457910c070a, State: Initialized, Role: FOLLOWER
I20260924 17:42:56.556383 22090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260924 17:42:56.556535 22090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:56.556591 22090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:56.556674 22090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:56.557585 22090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260924 17:42:56.557672 22090 leader_election.cc:302] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e6b8ca56b14da382741457910c070a; no voters:
I20260924 17:42:56.557858 22090 leader_election.cc:288] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:56.557878 22095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:56.558091 22095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [term 1 LEADER]: Becoming Leader. State: Replica: 29e6b8ca56b14da382741457910c070a, State: Running, Role: LEADER
I20260924 17:42:56.558234 22090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:56.558234 22095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260924 17:42:56.558700 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e6b8ca56b14da382741457910c070a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } } }
I20260924 17:42:56.558729 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e6b8ca56b14da382741457910c070a. Latest consensus state: current_term: 1 leader_uuid: "29e6b8ca56b14da382741457910c070a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e6b8ca56b14da382741457910c070a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } } }
I20260924 17:42:56.558866 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:56.558959 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a [sys.catalog]: This master's current role is: LEADER
W20260924 17:42:56.560194 22111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:56.560293 22111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:56.560397 22104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:56.560566 22104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:56.561964 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:56.556467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57505 (local address 127.0.98.62:43601)
0924 17:42:56.556703 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:56.556714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:56.556750 (+ 36us) server_negotiation.cc:407] Connection header received
0924 17:42:56.556859 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:56.556866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:56.556915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:56.557053 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:56.557242 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:56.558420 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:56.559595 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:56.560157 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:56.560281 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:56.561018 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:56.561022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:56.561023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:56.561270 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:56.561306 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:56.561320 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:56.561416 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:56.561702 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 17:42:56.561737 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260924 17:42:56.562139 22104 catalog_manager.cc:1347] Generated new cluster ID: 50fe993727cd4b43849dd1c2af964105
I20260924 17:42:56.562192 22104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:56.562908 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:56.570694 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:56.564327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44683 (local address 127.0.98.62:43601)
0924 17:42:56.564365 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:56.564367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:56.567234 (+ 2867us) server_negotiation.cc:407] Connection header received
0924 17:42:56.567304 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:56.567305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:56.567317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:56.567374 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:56.567527 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:56.568309 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:56.569288 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:56.569603 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:56.569680 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:56.570436 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:56.570437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:56.570438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:56.570443 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:56.570459 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:56.570464 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:56.570528 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:56.570599 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:42:56.570612 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:42:56.592293 22104 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:56.593073 22104 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:56.603379 22104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29e6b8ca56b14da382741457910c070a: Generated new TSK 0
I20260924 17:42:56.603637 22104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:56.606602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22022
W20260924 17:42:56.611526 22007 connection.cc:535] client connection to 127.0.98.62:43601 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43601): Connection reset by peer (error 104)
2026-09-24T17:42:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:56 dist-test-slave-d0pf krb5kdc[22121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:56 dist-test-slave-d0pf krb5kdc[22121](info): set up 2 sockets
Sep 24 17:42:56 dist-test-slave-d0pf krb5kdc[22121](info): commencing operation
krb5kdc: starting...
W20260924 17:42:57.655197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:57 dist-test-slave-d0pf krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:57.774231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:57.855675 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:57.855929 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:57.855971 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:57.857182 22134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:57.857255 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:57.857280 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:57.857301 22134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:57.857329 22134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:57.858865 22134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:57.859174 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:57.859417 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:57.864185 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:57.864362 22134 server_base.cc:1034] running on GCE node
W20260924 17:42:57.864197 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:57.864197 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:57.864730 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:57.865068 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:57.866248 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790271777866242 us; error 35 us; skew 500 ppm
Sep 24 17:42:57 dist-test-slave-d0pf krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:57.868845 22134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:57.869257 22134 webserver.cc:460] Webserver started at http://127.0.98.62:45477/ using document root <none> and password file <none>
I20260924 17:42:57.869496 22134 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:57.869565 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:57.869709 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:57.870729 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "933196aa11e3419595385e30e38c6ccb"
format_stamp: "Formatted at 2026-09-24 17:42:57 on dist-test-slave-d0pf"
server_key: "3c999e985242efa90a288fc6c82c5c1d"
server_key_iv: "a72ff9c3293777477c658fd1d062346f"
server_key_version: "encryptionkey@0"
I20260924 17:42:57.871151 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "933196aa11e3419595385e30e38c6ccb"
format_stamp: "Formatted at 2026-09-24 17:42:57 on dist-test-slave-d0pf"
server_key: "3c999e985242efa90a288fc6c82c5c1d"
server_key_iv: "a72ff9c3293777477c658fd1d062346f"
server_key_version: "encryptionkey@0"
I20260924 17:42:57.872438 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:42:57.873152 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:57.873317 22134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:42:57.873415 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "933196aa11e3419595385e30e38c6ccb"
format_stamp: "Formatted at 2026-09-24 17:42:57 on dist-test-slave-d0pf"
server_key: "3c999e985242efa90a288fc6c82c5c1d"
server_key_iv: "a72ff9c3293777477c658fd1d062346f"
server_key_version: "encryptionkey@0"
I20260924 17:42:57.873497 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:57.876786 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:57.878566 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:57.878726 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:57.882704 22134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42893
I20260924 17:42:57.882733 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42893 every 8 connection(s)
I20260924 17:42:57.883123 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:57.884194 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:57.886484 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: Bootstrap starting.
I20260924 17:42:57.887153 22202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:57.887444 22202 log.cc:824] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:57.888159 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: No bootstrap required, opened a new log
I20260924 17:42:57.889055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22134
I20260924 17:42:57.889178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:57.889348 392 external_mini_cluster.cc:1428] Setting key 16b3b4b27868c5832002a5ece2067637
I20260924 17:42:57.890322 22202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 17:42:57.890492 22202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:57.890555 22202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933196aa11e3419595385e30e38c6ccb, State: Initialized, Role: FOLLOWER
I20260924 17:42:57.891018 22202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 17:42:57.891176 22202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:57.891222 22202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:57.891275 22202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:57.891878 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:57.889916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57201 (local address 127.0.98.62:42893)
0924 17:42:57.890107 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:57.890115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:57.890139 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:42:57.890283 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:57.890288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:57.890312 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:57.890467 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:57.891089 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:57.891093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:57.891093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:57.891097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:57.891140 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:57.891153 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:57.891237 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:57.891353 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 17:42:57.891400 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 17:42:57.892102 22202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 17:42:57.892222 22202 leader_election.cc:302] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933196aa11e3419595385e30e38c6ccb; no voters:
I20260924 17:42:57.892386 22202 leader_election.cc:288] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:57.892447 22207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:57.892678 22202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:57.892681 22207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [term 1 LEADER]: Becoming Leader. State: Replica: 933196aa11e3419595385e30e38c6ccb, State: Running, Role: LEADER
I20260924 17:42:57.892907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:57.892915 22207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 17:42:57.893267 22207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933196aa11e3419595385e30e38c6ccb. Latest consensus state: current_term: 1 leader_uuid: "933196aa11e3419595385e30e38c6ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } } }
I20260924 17:42:57.893345 22207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:57.893359 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "933196aa11e3419595385e30e38c6ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933196aa11e3419595385e30e38c6ccb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } } }
I20260924 17:42:57.893446 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb [sys.catalog]: This master's current role is: LEADER
W20260924 17:42:57.894907 22227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:42:57.894975 22227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:42:57.895063 22216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:57.895249 22216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:57.896660 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:57.894466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47887 (local address 127.0.98.62:42893)
0924 17:42:57.894496 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:57.894498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:57.895170 (+ 672us) server_negotiation.cc:407] Connection header received
0924 17:42:57.895246 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:57.895247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:57.895253 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:57.895313 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:57.896397 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:57.896398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:57.896399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:42:57.896403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:57.896418 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:42:57.896422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:57.896453 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:57.896563 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:42:57.896578 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 17:42:57.896967 22216 catalog_manager.cc:1347] Generated new cluster ID: 0b300adf9a5b421da0fdec087830c437
I20260924 17:42:57.897013 22216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:57.918691 22216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:57.918888 22216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:57.922775 22216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933196aa11e3419595385e30e38c6ccb: Generated new TSK 0
I20260924 17:42:57.922968 22216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:57.929775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22134
2026-09-24T17:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:57 dist-test-slave-d0pf krb5kdc[22233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:57 dist-test-slave-d0pf krb5kdc[22233](info): set up 2 sockets
Sep 24 17:42:57 dist-test-slave-d0pf krb5kdc[22233](info): commencing operation
krb5kdc: starting...
W20260924 17:42:58.967159 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:42:59.086808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:42:59.169399 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:42:59.169674 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:42:59.169718 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:42:59.171096 22246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:42:59.171188 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:42:59.171216 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:42:59.171249 22246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:42:59.171273 22246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:42:59.172847 22246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:42:59.173141 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:42:59.173396 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:42:59.178465 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:59.178472 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:42:59.178472 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:42:59.178709 22246 server_base.cc:1034] running on GCE node
I20260924 17:42:59.179006 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:42:59.179368 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:42:59.180524 22246 hybrid_clock.cc:648] HybridClock initialized: now 1790271779180503 us; error 39 us; skew 500 ppm
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:42:59.183166 22246 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:42:59.183645 22246 webserver.cc:460] Webserver started at http://127.0.98.62:36083/ using document root <none> and password file <none>
I20260924 17:42:59.183892 22246 fs_manager.cc:359] Metadata directory not provided
I20260924 17:42:59.183964 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:42:59.184114 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:42:59.185271 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "0d4f6d08b9b9486e9f4435f628c86a24"
format_stamp: "Formatted at 2026-09-24 17:42:59 on dist-test-slave-d0pf"
server_key: "e60d0597f00f3dbd09e4fb6abc5a7a79"
server_key_iv: "33823b3528af867c0db8216d9e3d8c93"
server_key_version: "encryptionkey@0"
I20260924 17:42:59.185685 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d4f6d08b9b9486e9f4435f628c86a24"
format_stamp: "Formatted at 2026-09-24 17:42:59 on dist-test-slave-d0pf"
server_key: "e60d0597f00f3dbd09e4fb6abc5a7a79"
server_key_iv: "33823b3528af867c0db8216d9e3d8c93"
server_key_version: "encryptionkey@0"
I20260924 17:42:59.187189 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:42:59.188105 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:42:59.188442 22246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:42:59.188591 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0d4f6d08b9b9486e9f4435f628c86a24"
format_stamp: "Formatted at 2026-09-24 17:42:59 on dist-test-slave-d0pf"
server_key: "e60d0597f00f3dbd09e4fb6abc5a7a79"
server_key_iv: "33823b3528af867c0db8216d9e3d8c93"
server_key_version: "encryptionkey@0"
I20260924 17:42:59.188678 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:42:59.203197 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:42:59.205067 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:42:59.205255 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:42:59.209321 22246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41975
I20260924 17:42:59.209359 22313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41975 every 8 connection(s)
I20260924 17:42:59.209738 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:42:59.210654 22314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:42:59.211889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22246
I20260924 17:42:59.212033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:42:59.212198 392 external_mini_cluster.cc:1428] Setting key cc272fbdda25179723ced14096705053
I20260924 17:42:59.213395 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24: Bootstrap starting.
I20260924 17:42:59.214208 22314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24: Neither blocks nor log segments found. Creating new log.
I20260924 17:42:59.214547 22314 log.cc:824] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24: Log is configured to *not* fsync() on all Append() calls
I20260924 17:42:59.215427 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24: No bootstrap required, opened a new log
I20260924 17:42:59.217908 22314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } }
I20260924 17:42:59.218205 22314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:42:59.218254 22314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4f6d08b9b9486e9f4435f628c86a24, State: Initialized, Role: FOLLOWER
I20260924 17:42:59.218406 22314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } }
I20260924 17:42:59.218521 22314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:42:59.218559 22314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:42:59.218605 22314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:42:59.219578 22314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } }
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:42:59.219679 22314 leader_election.cc:302] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d4f6d08b9b9486e9f4435f628c86a24; no voters:
I20260924 17:42:59.219892 22314 leader_election.cc:288] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:42:59.219923 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:42:59.220184 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [term 1 LEADER]: Becoming Leader. State: Replica: 0d4f6d08b9b9486e9f4435f628c86a24, State: Running, Role: LEADER
I20260924 17:42:59.220327 22314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:42:59.220381 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } }
I20260924 17:42:59.220821 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d4f6d08b9b9486e9f4435f628c86a24. Latest consensus state: current_term: 1 leader_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } } }
I20260924 17:42:59.221076 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:59.220997 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4f6d08b9b9486e9f4435f628c86a24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41975 } } }
I20260924 17:42:59.221299 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24 [sys.catalog]: This master's current role is: LEADER
I20260924 17:42:59.221632 22328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:42:59.222188 22328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:42:59.223870 22328 catalog_manager.cc:1347] Generated new cluster ID: 7da26879e20244ce9a53274e6b9da1b9
I20260924 17:42:59.223927 22328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:42:59.224500 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:59.212764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58151 (local address 127.0.98.62:41975)
0924 17:42:59.212935 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:59.212944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:59.212976 (+ 32us) server_negotiation.cc:407] Connection header received
0924 17:42:59.213384 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:59.213390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:59.213435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:59.213602 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:59.214448 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:59.215514 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:59.216644 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:59.217119 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:59.217212 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:59.220313 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:59.220317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:59.220317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:59.220332 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:59.222607 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:59.223076 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:59.223079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:59.223080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:59.223192 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:59.223454 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:59.223456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:59.223456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:59.223727 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:59.223889 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:59.224184 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 17:42:59.224328 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260924 17:42:59.227927 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:42:59.237939 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:42:59.229296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43503 (local address 127.0.98.62:41975)
0924 17:42:59.229342 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:42:59.229345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:42:59.231256 (+ 1911us) server_negotiation.cc:407] Connection header received
0924 17:42:59.231331 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:42:59.231332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:42:59.231344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:42:59.231401 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:42:59.232120 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:59.232638 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:59.233378 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:42:59.233617 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:42:59.233696 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:42:59.235008 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:42:59.235010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:42:59.235010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:42:59.235018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:42:59.236509 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:59.236911 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:59.236912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:59.236912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:59.236989 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:42:59.237211 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:42:59.237212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:42:59.237212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:42:59.237386 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:42:59.237487 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 17:42:59.237657 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 17:42:59.237765 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 17:42:59.242123 22328 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:42:59.242652 22328 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:42:59.250414 22328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d4f6d08b9b9486e9f4435f628c86a24: Generated new TSK 0
I20260924 17:42:59.250643 22328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:42:59.254196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22246
2026-09-24T17:42:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22345](info): set up 2 sockets
Sep 24 17:42:59 dist-test-slave-d0pf krb5kdc[22345](info): commencing operation
krb5kdc: starting...
W20260924 17:43:00.287279 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:43:00 dist-test-slave-d0pf krb5kdc[22345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:43:00.407572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42471
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:00.487200 22358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:00.487408 22358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:00.487432 22358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:00.488680 22358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:00.488726 22358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:00.488740 22358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:00.488754 22358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:00.488765 22358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:00.490192 22358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:00.490384 22358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:00.490556 22358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:00.495357 22366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:00.495352 22364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:00.495352 22363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:00.495692 22358 server_base.cc:1034] running on GCE node
I20260924 17:43:00.496001 22358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:00.496294 22358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:00.497462 22358 hybrid_clock.cc:648] HybridClock initialized: now 1790271780497448 us; error 31 us; skew 500 ppm
Sep 24 17:43:00 dist-test-slave-d0pf krb5kdc[22345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:00.500102 22358 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:43:00.500516 22358 webserver.cc:460] Webserver started at http://127.0.98.62:33313/ using document root <none> and password file <none>
I20260924 17:43:00.500746 22358 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:00.500815 22358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:00.500973 22358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:00.502112 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "0f0137ea3fa942f9a748551cbb7f38d9"
format_stamp: "Formatted at 2026-09-24 17:43:00 on dist-test-slave-d0pf"
server_key: "bf65fb3326b240bb0861609c58120c64"
server_key_iv: "67b0c963ee70c9a4aa9d989a65976872"
server_key_version: "encryptionkey@0"
I20260924 17:43:00.502476 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f0137ea3fa942f9a748551cbb7f38d9"
format_stamp: "Formatted at 2026-09-24 17:43:00 on dist-test-slave-d0pf"
server_key: "bf65fb3326b240bb0861609c58120c64"
server_key_iv: "67b0c963ee70c9a4aa9d989a65976872"
server_key_version: "encryptionkey@0"
I20260924 17:43:00.503903 22358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 17:43:00.504655 22373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:00.504827 22358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:00.504942 22358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "0f0137ea3fa942f9a748551cbb7f38d9"
format_stamp: "Formatted at 2026-09-24 17:43:00 on dist-test-slave-d0pf"
server_key: "bf65fb3326b240bb0861609c58120c64"
server_key_iv: "67b0c963ee70c9a4aa9d989a65976872"
server_key_version: "encryptionkey@0"
I20260924 17:43:00.505018 22358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:00.516351 22358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:00.518858 22358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:00.519088 22358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:00.522922 22358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42471
I20260924 17:43:00.522982 22425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42471 every 8 connection(s)
I20260924 17:43:00.523406 22358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:00.524228 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:00.526402 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9: Bootstrap starting.
I20260924 17:43:00.526921 22426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:00.527235 22426 log.cc:824] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:00.527872 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9: No bootstrap required, opened a new log
I20260924 17:43:00.529352 22426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } }
I20260924 17:43:00.529485 22426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:00.529508 22426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0137ea3fa942f9a748551cbb7f38d9, State: Initialized, Role: FOLLOWER
I20260924 17:43:00.529624 22426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } }
I20260924 17:43:00.529676 22426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:00.529696 22426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:00.529723 22426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:00.530169 22426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } }
I20260924 17:43:00.530252 22426 leader_election.cc:302] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f0137ea3fa942f9a748551cbb7f38d9; no voters:
I20260924 17:43:00.530367 22426 leader_election.cc:288] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:00.530421 22429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:00.530610 22429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0137ea3fa942f9a748551cbb7f38d9, State: Running, Role: LEADER
I20260924 17:43:00.530637 22426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:00.530750 22429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } }
I20260924 17:43:00.531384 22430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } } }
I20260924 17:43:00.531464 22430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:00.531419 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f0137ea3fa942f9a748551cbb7f38d9. Latest consensus state: current_term: 1 leader_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0137ea3fa942f9a748551cbb7f38d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42471 } } }
I20260924 17:43:00.531607 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:00.531983 22438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:00.532229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22358
I20260924 17:43:00.532353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:43:00.532586 22438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:00.532603 392 external_mini_cluster.cc:1428] Setting key 954fd1190c986a91224b4ab67238264e
I20260924 17:43:00.534253 22438 catalog_manager.cc:1347] Generated new cluster ID: dc357bbe9328410a9c6f46c71acb5a81
I20260924 17:43:00.534351 22438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:00.537809 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:00.533080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41769 (local address 127.0.98.62:42471)
0924 17:43:00.533267 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:00.533278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:00.533477 (+ 199us) server_negotiation.cc:407] Connection header received
0924 17:43:00.533599 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:00.533607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:00.533663 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:00.533811 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:00.534078 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:00.535088 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:00.535904 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:00.536343 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:00.536462 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:00.537159 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:00.537162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:00.537163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:00.537340 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:00.537370 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:00.537381 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:00.537453 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:00.537524 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:43:00.537589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260924 17:43:00.538699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:00.546227 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:00.543305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34843 (local address 127.0.98.62:42471)
0924 17:43:00.543341 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:00.543342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:00.543572 (+ 230us) server_negotiation.cc:407] Connection header received
0924 17:43:00.543635 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:00.543636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:00.543648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:00.543711 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:00.543855 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:00.544310 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:00.545005 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:00.545205 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:00.545286 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:00.545958 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:00.545960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:00.545960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:00.545965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:00.545978 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:00.545982 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:00.546042 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:00.546106 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 17:43:00.546130 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:00.554525 22438 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:00.555105 22438 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:00.560843 22438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f0137ea3fa942f9a748551cbb7f38d9: Generated new TSK 0
I20260924 17:43:00.561029 22438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:00.563899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22358
2026-09-24T17:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:00 dist-test-slave-d0pf krb5kdc[22457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:00 dist-test-slave-d0pf krb5kdc[22457](info): set up 2 sockets
Sep 24 17:43:00 dist-test-slave-d0pf krb5kdc[22457](info): commencing operation
krb5kdc: starting...
W20260924 17:43:01.591344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:43:01 dist-test-slave-d0pf krb5kdc[22457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:43:01.710843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:01.790822 22470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:01.791064 22470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:01.791105 22470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:01.792276 22470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:01.792318 22470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:01.792332 22470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:01.792371 22470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:01.792395 22470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:01.793855 22470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:01.794100 22470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:01.794328 22470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:01.799235 22476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:01.799244 22475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:01.799398 22478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:01.799399 22470 server_base.cc:1034] running on GCE node
I20260924 17:43:01.799764 22470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:01.800045 22470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:01.801216 22470 hybrid_clock.cc:648] HybridClock initialized: now 1790271781801196 us; error 33 us; skew 500 ppm
Sep 24 17:43:01 dist-test-slave-d0pf krb5kdc[22457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:01.809994 22470 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:43:01.810472 22470 webserver.cc:460] Webserver started at http://127.0.98.62:43801/ using document root <none> and password file <none>
I20260924 17:43:01.810712 22470 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:01.810784 22470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:01.810940 22470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:01.812072 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "f17ae3138f864f378e4d0b06089dc3a1"
format_stamp: "Formatted at 2026-09-24 17:43:01 on dist-test-slave-d0pf"
server_key: "487c776c9239297e61386f7fd0fe4daa"
server_key_iv: "ba9e0e4b8fdee0909df2fd82ac1b8d94"
server_key_version: "encryptionkey@0"
I20260924 17:43:01.812443 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "f17ae3138f864f378e4d0b06089dc3a1"
format_stamp: "Formatted at 2026-09-24 17:43:01 on dist-test-slave-d0pf"
server_key: "487c776c9239297e61386f7fd0fe4daa"
server_key_iv: "ba9e0e4b8fdee0909df2fd82ac1b8d94"
server_key_version: "encryptionkey@0"
I20260924 17:43:01.813828 22470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:01.814568 22485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:01.814741 22470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:01.814841 22470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "f17ae3138f864f378e4d0b06089dc3a1"
format_stamp: "Formatted at 2026-09-24 17:43:01 on dist-test-slave-d0pf"
server_key: "487c776c9239297e61386f7fd0fe4daa"
server_key_iv: "ba9e0e4b8fdee0909df2fd82ac1b8d94"
server_key_version: "encryptionkey@0"
I20260924 17:43:01.814924 22470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:01.818177 22470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:01.820276 22470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:01.820438 22470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:01.824345 22470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38917
I20260924 17:43:01.824378 22537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38917 every 8 connection(s)
I20260924 17:43:01.824738 22470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:01.825385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22470
I20260924 17:43:01.825527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:43:01.825762 392 external_mini_cluster.cc:1428] Setting key 62565d46b81303544b124555fad46780
I20260924 17:43:01.826017 22538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:01.828416 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:01.826244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57127 (local address 127.0.98.62:38917)
0924 17:43:01.826429 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:01.826439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:01.826473 (+ 34us) server_negotiation.cc:407] Connection header received
0924 17:43:01.827134 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:01.827141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:01.827168 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:01.827311 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:01.828012 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:01.828018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:01.828018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:01.828027 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:01.828062 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:01.828073 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:01.828149 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:01.828221 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 17:43:01.828262 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260924 17:43:01.829458 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:01.829469 22538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1: Bootstrap starting.
I20260924 17:43:01.830073 22538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:01.830413 22538 log.cc:824] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:01.831265 22538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1: No bootstrap required, opened a new log
I20260924 17:43:01.832054 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:01.830842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35329 (local address 127.0.98.62:38917)
0924 17:43:01.830891 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:01.830893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:01.831109 (+ 216us) server_negotiation.cc:407] Connection header received
0924 17:43:01.831172 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:01.831173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:01.831178 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:01.831245 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:01.831848 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:01.831850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:01.831850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:01.831854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:01.831870 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:01.831876 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:01.831941 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:01.831981 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 17:43:01.831993 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:43:01.833523 22538 raft_consensus.cc:348] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } }
I20260924 17:43:01.833703 22538 raft_consensus.cc:374] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:01.833741 22538 raft_consensus.cc:729] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17ae3138f864f378e4d0b06089dc3a1, State: Initialized, Role: FOLLOWER
I20260924 17:43:01.833828 22538 consensus_queue.cc:260] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } }
I20260924 17:43:01.833889 22538 raft_consensus.cc:388] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:01.833910 22538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:01.833941 22538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:01.834442 22538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } }
I20260924 17:43:01.834525 22538 leader_election.cc:302] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f17ae3138f864f378e4d0b06089dc3a1; no voters:
I20260924 17:43:01.834661 22538 leader_election.cc:288] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:01.834741 22548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:01.834908 22538 sys_catalog.cc:564] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:01.834935 22548 raft_consensus.cc:686] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [term 1 LEADER]: Becoming Leader. State: Replica: f17ae3138f864f378e4d0b06089dc3a1, State: Running, Role: LEADER
I20260924 17:43:01.835125 22548 consensus_queue.cc:237] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } }
I20260924 17:43:01.835534 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f17ae3138f864f378e4d0b06089dc3a1. Latest consensus state: current_term: 1 leader_uuid: "f17ae3138f864f378e4d0b06089dc3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } }
I20260924 17:43:01.835526 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f17ae3138f864f378e4d0b06089dc3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17ae3138f864f378e4d0b06089dc3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } }
I20260924 17:43:01.835644 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:01.835649 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:01.835906 22557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:01.836409 22557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:01.837903 22557 catalog_manager.cc:1347] Generated new cluster ID: 13e9e62a0ade4625a19e080e27f8e37e
I20260924 17:43:01.837955 22557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:01.855790 22557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:01.855979 22557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:01.862659 22557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f17ae3138f864f378e4d0b06089dc3a1: Generated new TSK 0
I20260924 17:43:01.862835 22557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:01.871390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22470
2026-09-24T17:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1307 ms)
[----------] 18 tests from AuthTokenIssuingTest (23796 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:01 dist-test-slave-d0pf krb5kdc[22569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:01 dist-test-slave-d0pf krb5kdc[22569](info): set up 2 sockets
Sep 24 17:43:01 dist-test-slave-d0pf krb5kdc[22569](info): commencing operation
krb5kdc: starting...
W20260924 17:43:02.907469 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:43:02 dist-test-slave-d0pf krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:43:03.026281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:03.106953 22582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:03.107197 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:03.107215 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:03.107234 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:03.108317 22582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:03.108352 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:03.108366 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:03.108378 22582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:03.108390 22582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:03.109839 22582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:03.110028 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:03.110199 22582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:03.114948 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.115011 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.114944 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:03.115186 22582 server_base.cc:1034] running on GCE node
I20260924 17:43:03.115461 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:03.115784 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:03.116971 22582 hybrid_clock.cc:648] HybridClock initialized: now 1790271783116956 us; error 38 us; skew 500 ppm
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:03.119793 22582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:43:03.120278 22582 webserver.cc:460] Webserver started at http://127.0.98.62:34835/ using document root <none> and password file <none>
I20260924 17:43:03.120523 22582 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:03.120592 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:03.120754 22582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:03.121797 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "ed74d0a504d04f4f98262647ef8e29ef"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "b1842918788dde46632f079b45a9c938"
server_key_iv: "ef00f514865fb3007ca662d4d8cd2864"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.122154 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed74d0a504d04f4f98262647ef8e29ef"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "b1842918788dde46632f079b45a9c938"
server_key_iv: "ef00f514865fb3007ca662d4d8cd2864"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.123533 22582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:03.124320 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:03.124531 22582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:03.124631 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "ed74d0a504d04f4f98262647ef8e29ef"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "b1842918788dde46632f079b45a9c938"
server_key_iv: "ef00f514865fb3007ca662d4d8cd2864"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.124739 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:03.136389 22582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:03.138682 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:03.138854 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:03.142759 22582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34895
I20260924 17:43:03.142791 22649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34895 every 8 connection(s)
I20260924 17:43:03.143204 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:03.144243 22650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:03.145781 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:03.151101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22582
I20260924 17:43:03.151233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:43:03.151427 392 external_mini_cluster.cc:1428] Setting key 9bae033252a7f46c49052db16f83e312
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:03.155544 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.146468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34895 (local address 127.0.0.1:32890)
0924 17:43:03.146633 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.146670 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.147177 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.147404 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.147406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.147652 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.147810 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.147814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.148798 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.148799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.149464 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.149466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.149949 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.149950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.150004 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.150454 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.150462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.152111 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.154023 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.154029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.154032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.154221 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.154483 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.154484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.154484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.154567 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.154924 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.154930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.154998 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.155124 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.155361 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":94,"thread_start_us":92,"threads_started":1}
I20260924 17:43:03.155546 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.146511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.0.98.62:34895)
0924 17:43:03.146664 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.146673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.146742 (+ 69us) server_negotiation.cc:407] Connection header received
0924 17:43:03.147301 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.147303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.147318 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.147397 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.147910 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.148708 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.149534 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.149857 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.149940 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.152221 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.152228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.152229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.152235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.153915 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.154320 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.154323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.154323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.154408 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.154632 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.154633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.154633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.154851 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.154947 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.155148 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.155296 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 17:43:03.156302 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:03.156656 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.156539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41860)
0924 17:43:03.156586 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.156632 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:43:03.156793 22598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
I20260924 17:43:03.157222 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.157130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41866)
0924 17:43:03.157177 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.157200 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:43:03.157325 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39253: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 17:43:03.175985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39253
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:03.206557 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } attempt: 1
I20260924 17:43:03.207120 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.206937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41882)
0924 17:43:03.207010 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.207079 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 17:43:03.207582 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.207469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41888)
0924 17:43:03.207523 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.207562 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:43:03.207756 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39253: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
W20260924 17:43:03.257879 22657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:03.258106 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:03.258124 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:03.258176 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:03.259356 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:03.259399 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:03.259413 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:03.259452 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:03.259480 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:03.261083 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:03.261355 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:03.261591 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:03.266440 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.266456 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.266582 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:03.266575 22657 server_base.cc:1034] running on GCE node
I20260924 17:43:03.266896 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:03.267213 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:03.268386 22657 hybrid_clock.cc:648] HybridClock initialized: now 1790271783268371 us; error 28 us; skew 500 ppm
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:03.270920 22657 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 17:43:03.271447 22657 webserver.cc:460] Webserver started at http://127.0.98.61:33173/ using document root <none> and password file <none>
I20260924 17:43:03.271656 22657 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:03.271698 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:03.271790 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:03.272835 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data/instance:
uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "3baf62ee810c970cc9362d4e3c67009f"
server_key_iv: "62340fd9fdcef53f076a4397152f42b3"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.273149 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal/instance:
uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "3baf62ee810c970cc9362d4e3c67009f"
server_key_iv: "62340fd9fdcef53f076a4397152f42b3"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.274478 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:03.275277 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:03.275491 22657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:03.275594 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "3baf62ee810c970cc9362d4e3c67009f"
server_key_iv: "62340fd9fdcef53f076a4397152f42b3"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.275679 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:03.275945 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } attempt: 2
I20260924 17:43:03.276332 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.276219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41902)
0924 17:43:03.276273 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.276306 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:43:03.276680 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.276581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41918)
0924 17:43:03.276623 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.276653 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 17:43:03.276764 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39253: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39253: connect: Connection refused (error 111)
I20260924 17:43:03.288194 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:03.290364 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:03.290529 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:03.295042 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39253
I20260924 17:43:03.295132 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39253 every 8 connection(s)
I20260924 17:43:03.295477 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
I20260924 17:43:03.296478 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:03.298050 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:03.300833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22657
I20260924 17:43:03.300947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-1/wal/instance
I20260924 17:43:03.301136 392 external_mini_cluster.cc:1428] Setting key 118548c4ab26bd26e31c0764164d2ab5
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:03.307977 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.298854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.0.98.62:34895)
0924 17:43:03.298922 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.298924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.299127 (+ 203us) server_negotiation.cc:407] Connection header received
0924 17:43:03.299597 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.299599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.299612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.299685 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.300302 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.300880 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.302021 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.302350 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.302433 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.304998 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.304999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.304999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.305008 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.306511 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.306948 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.306954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.306955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.307010 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.307313 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.307314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.307314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.307430 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.307504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.307794 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.307882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:03.308178 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.298781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34895 (local address 127.0.0.1:32894)
0924 17:43:03.298952 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.298994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.299509 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.299741 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.299744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.300007 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.300221 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.300225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.300991 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.300992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.301929 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.301931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.302489 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.302490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.302661 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.303248 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.303261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.304893 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.306606 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.306613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.306615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.306860 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.307113 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.307114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.307115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.307235 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.307496 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.307503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.307652 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.307761 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.307930 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260924 17:43:03.308797 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:03.316919 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.309177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.0.98.61:39253)
0924 17:43:03.309352 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.309356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.309405 (+ 49us) server_negotiation.cc:407] Connection header received
0924 17:43:03.309446 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.309447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.309459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.309518 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.309781 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.310552 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.311376 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.311799 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.311891 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.313957 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.313965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.313966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.313972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.315555 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.316002 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.316005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.316005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.316058 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.316275 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.316275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.316276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.316433 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.316498 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.316755 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.316835 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260924 17:43:03.317083 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.309099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41920)
0924 17:43:03.309342 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.309353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.309408 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.309536 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.309537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.309671 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.309721 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.309722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.310624 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.310624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.311297 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.311300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.311890 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.311891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.311956 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.312337 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.312343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.313875 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.315677 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.315678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.315679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.315919 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.316138 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.316139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.316140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.316212 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.316508 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.316509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.316564 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.316750 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.316915 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":24}
I20260924 17:43:03.317907 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:03.318279 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.318139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42806)
0924 17:43:03.318190 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.318245 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:43:03.318437 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
I20260924 17:43:03.318926 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.318816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42820)
0924 17:43:03.318868 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.318901 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:43:03.319101 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 17:43:03.330108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45885
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:03.367914 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } attempt: 3
I20260924 17:43:03.368331 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } attempt: 1
I20260924 17:43:03.368749 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.368591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42836)
0924 17:43:03.368685 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.368715 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:43:03.369109 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.369031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42840)
0924 17:43:03.369067 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.369087 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 17:43:03.369220 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:03.375208 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.368324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41930)
0924 17:43:03.368382 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.368395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.368451 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.368641 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.368643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.368829 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.368915 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.368917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.369666 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.369668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.370182 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.370184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.370578 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.370579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.370661 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.371164 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.371169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.372547 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.374150 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.374151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.374152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.374323 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.374536 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.374537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.374538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.374603 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.374854 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.374855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.374900 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.374990 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.375109 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 17:43:03.375207 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.368361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.98.61:39253)
0924 17:43:03.368404 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.368406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.368436 (+ 30us) server_negotiation.cc:407] Connection header received
0924 17:43:03.368528 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.368529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.368548 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.368608 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.368994 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.369577 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.370280 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.370503 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.370580 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.372652 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.372659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.372659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.372674 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.374052 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.374410 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.374411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.374411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.374468 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.374667 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.374668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.374669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.374784 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.374863 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.375013 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.375111 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:03.375675 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:03.376046 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.375954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42856)
0924 17:43:03.376000 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.376027 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:43:03.376361 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.376273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42862)
0924 17:43:03.376316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.376340 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:43:03.376454 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
I20260924 17:43:03.407583 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } attempt: 1
I20260924 17:43:03.408027 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.407891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42876)
0924 17:43:03.407956 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.407995 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:43:03.408363 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.408270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42890)
0924 17:43:03.408314 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.408337 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 17:43:03.408447 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
W20260924 17:43:03.413489 22732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:03.413687 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:03.413704 22732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:03.413756 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:03.414896 22732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:03.414935 22732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:03.414949 22732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:03.414987 22732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:03.415014 22732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:03.416554 22732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34895,127.0.98.61:39253,127.0.98.60:45885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:03.416834 22732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:03.417080 22732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:03.422348 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.422295 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:03.422317 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:03.422350 22732 server_base.cc:1034] running on GCE node
I20260924 17:43:03.422827 22732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:03.423202 22732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:03.424376 22732 hybrid_clock.cc:648] HybridClock initialized: now 1790271783424368 us; error 38 us; skew 500 ppm
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:03.427067 22732 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 17:43:03.427567 22732 webserver.cc:460] Webserver started at http://127.0.98.60:35061/ using document root <none> and password file <none>
I20260924 17:43:03.427846 22732 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:03.427891 22732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:03.428043 22732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:03.429136 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data/instance:
uuid: "3296d6bc9c5f492586665c6d15b5a1e5"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "d8eb3a740608b176413f02a411202079"
server_key_iv: "d6ad4eb04cba8f594a444eb1635089e4"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.429493 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal/instance:
uuid: "3296d6bc9c5f492586665c6d15b5a1e5"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "d8eb3a740608b176413f02a411202079"
server_key_iv: "d6ad4eb04cba8f594a444eb1635089e4"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.430837 22732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:03.431640 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:03.431841 22732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:43:03.431926 22732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "3296d6bc9c5f492586665c6d15b5a1e5"
format_stamp: "Formatted at 2026-09-24 17:43:03 on dist-test-slave-d0pf"
server_key: "d8eb3a740608b176413f02a411202079"
server_key_iv: "d6ad4eb04cba8f594a444eb1635089e4"
server_key_version: "encryptionkey@0"
I20260924 17:43:03.431979 22732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:03.437350 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } attempt: 2
I20260924 17:43:03.437786 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.437647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42902)
0924 17:43:03.437710 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.437751 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:43:03.438098 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.437995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42916)
0924 17:43:03.438041 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.438067 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:43:03.438218 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45885: connect: Connection refused (error 111)
I20260924 17:43:03.449146 22732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:03.451351 22732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:03.451498 22732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:03.455492 22732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45885
I20260924 17:43:03.455554 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45885 every 8 connection(s)
I20260924 17:43:03.455920 22732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
I20260924 17:43:03.456993 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:03.458595 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:03.464921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 22732
I20260924 17:43:03.465049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271516406493-392-0/minicluster-data/master-2/wal/instance
I20260924 17:43:03.465240 392 external_mini_cluster.cc:1428] Setting key f2c1105e2c229b5c6b15288e3b0a0a53
I20260924 17:43:03.468797 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.459421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.62:34895)
0924 17:43:03.459529 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.459531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.459710 (+ 179us) server_negotiation.cc:407] Connection header received
0924 17:43:03.460230 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.460232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.460243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.460302 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.461010 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.461783 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.462965 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.463221 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.463278 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.465637 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.465638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.465638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.465646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.467315 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.467810 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.467811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.467811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.467871 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.468148 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.468149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.468149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.468294 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.468370 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.468615 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.468699 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 17:43:03.468884 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.459347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34895 (local address 127.0.0.1:32908)
0924 17:43:03.459564 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.459609 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.460121 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.460377 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.460380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.460640 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.460895 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.460902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.461893 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.461895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.462873 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.462876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.463311 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.463312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.463421 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.463869 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.463878 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.465341 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.467392 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.467398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.467400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.467714 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.467947 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.467948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.467948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.468082 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.468363 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.468367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.468486 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.468568 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.468693 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260924 17:43:03.469461 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:03.475184 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.465642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.0.98.60:45885)
0924 17:43:03.465824 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.465828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.465917 (+ 89us) server_negotiation.cc:407] Connection header received
0924 17:43:03.465968 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.465970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.465985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.466063 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.466918 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.467892 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.469009 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.469519 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.469617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.471979 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.471988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.471988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.471995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.473926 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.474286 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.474289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.474290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.474346 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.474541 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.474542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.474542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.474705 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.474867 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.474949 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.475053 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260924 17:43:03.476789 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.469754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39253 (local address 127.0.0.1:41934)
0924 17:43:03.469811 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.469822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.469857 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.469999 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.470000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.470160 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.470221 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.470222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.470837 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.470838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.471416 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.471418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.471893 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.471894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.471963 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.472392 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.472397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.473944 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.475735 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.475736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.475736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.475980 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.476185 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.476186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.476186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.476272 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.476493 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.476494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.476527 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.476577 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.476723 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:43:03.476918 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.469846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.0.98.61:39253)
0924 17:43:03.469896 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.469898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.469906 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:03.469929 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.469930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.469939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.469995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.470288 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.470755 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.471496 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.471810 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.471894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.474031 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.474033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.474033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.474040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.475642 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.476053 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.476054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.476055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.476118 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.476338 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.476338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.476339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.476443 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.476486 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.476711 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.476800 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:03.477281 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:03.483605 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } attempt: 2
I20260924 17:43:03.484748 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.476084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.62:34895)
0924 17:43:03.476139 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.476141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.476259 (+ 118us) server_negotiation.cc:407] Connection header received
0924 17:43:03.476286 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.476288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.476299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.476354 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.477232 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.477856 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.478608 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.478912 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.478965 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.481682 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.481684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.481684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.481693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.483455 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.483865 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.483866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.483866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.483928 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.484133 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.484134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.484134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.484286 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.484501 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.484538 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.484627 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:43:03.484777 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.477541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42934)
0924 17:43:03.477619 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.477707 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.477767 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.477885 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.477886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.478031 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.478059 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.478060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.478843 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.478844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.479619 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.479621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.480055 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.480056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.480123 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.480502 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.480507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.481960 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.483688 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.483689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.483690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.483890 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.484082 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.484083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.484083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.484156 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.484476 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.484476 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.484510 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.484583 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.484679 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 17:43:03.485075 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.477589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.98.60:45885)
0924 17:43:03.477703 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.477706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.477751 (+ 45us) server_negotiation.cc:407] Connection header received
0924 17:43:03.477825 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.477825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.477837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.477884 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.478140 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.478768 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.479772 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.479973 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.480035 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.482071 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.482072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.482073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.482079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.483620 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.483967 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.483968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.483969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.484017 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.484224 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.484224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.484225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.484390 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.484454 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.484931 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.484999 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 17:43:03.487859 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: Bootstrap starting.
I20260924 17:43:03.488672 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: Neither blocks nor log segments found. Creating new log.
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:03.488998 22800 log.cc:824] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:03.490163 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: No bootstrap required, opened a new log
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:03.492452 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.483871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42950)
0924 17:43:03.483933 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.483946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.483985 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.484320 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.484322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.484482 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.484525 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.484527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.485491 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.485492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.486304 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.486306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.487449 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.487450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.487510 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.487944 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.487949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.489174 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.491219 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.491220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.491221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.491431 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.491692 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.491694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.491694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.491763 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.492097 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.492099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.492153 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.492234 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.492328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:43:03.492491 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.484005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.0.98.60:45885)
0924 17:43:03.484195 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.484197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.484204 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:43:03.484221 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.484222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.484232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.484300 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.484604 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.485397 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.487172 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.487371 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.487451 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.489262 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.489264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.489264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.489271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.491126 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.491526 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.491527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.491528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.491600 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.491841 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.491842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.491842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.492008 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.492087 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.492271 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.492351 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260924 17:43:03.492339 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.492538 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296d6bc9c5f492586665c6d15b5a1e5, State: Initialized, Role: FOLLOWER
I20260924 17:43:03.492623 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.493019 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:03.493960 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.485343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.0.98.61:39253)
0924 17:43:03.485390 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.485392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.485431 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:43:03.485497 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.485498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.485508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.485564 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.486164 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.486620 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.487615 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.487893 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.487937 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.490806 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.490808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.490808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.490816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.492633 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.493020 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.493021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.493021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.493074 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.493279 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.493280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.493281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.493399 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.493582 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.493619 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.493857 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:43:03.494396 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: Bootstrap starting.
I20260924 17:43:03.494747 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.494869 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.495190 22650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: Neither blocks nor log segments found. Creating new log.
W20260924 17:43:03.495522 22818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 17:43:03.495539 22650 log.cc:824] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: Log is configured to *not* fsync() on all Append() calls
W20260924 17:43:03.495625 22818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:03.496394 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: No bootstrap required, opened a new log
I20260924 17:43:03.496733 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:03.497694 22624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ed74d0a504d04f4f98262647ef8e29ef"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32918
I20260924 17:43:03.495198 22732 master_runner.cc:186] Error getting master registration for 127.0.98.62:34895: OK, instance_id { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" instance_seqno: 1790271783141658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:43:03.498109 22732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:43:03.498355 22650 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.498484 22650 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74d0a504d04f4f98262647ef8e29ef, State: Initialized, Role: FOLLOWER
I20260924 17:43:03.498569 22650 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.498878 22650 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:03.498849 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.498924 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 17:43:03.498924 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.499001 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:03.499847 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.500092 22624 leader_election.cc:288] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [CANDIDATE]: Term 1 election: Requested vote from peers a78d1f19b5d944a0ba5c1f8cd5ddf372 (127.0.98.61:39253), 3296d6bc9c5f492586665c6d15b5a1e5 (127.0.98.60:45885)
I20260924 17:43:03.500399 22774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed74d0a504d04f4f98262647ef8e29ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3296d6bc9c5f492586665c6d15b5a1e5"
I20260924 17:43:03.500532 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [term 0 FOLLOWER]: Advancing to term 1
W20260924 17:43:03.501278 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:03.501333 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:03.501442 22774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed74d0a504d04f4f98262647ef8e29ef in term 1.
I20260924 17:43:03.502002 22599 leader_election.cc:302] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296d6bc9c5f492586665c6d15b5a1e5, ed74d0a504d04f4f98262647ef8e29ef; no voters:
I20260924 17:43:03.502157 22821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:03.502312 22821 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [term 1 LEADER]: Becoming Leader. State: Replica: ed74d0a504d04f4f98262647ef8e29ef, State: Running, Role: LEADER
I20260924 17:43:03.502404 22821 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.502816 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed74d0a504d04f4f98262647ef8e29ef. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.502892 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:03.503885 22582 master_runner.cc:186] Error getting master registration for 127.0.98.61:39253: OK, instance_id { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" instance_seqno: 1790271783293593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:43:03.506227 22582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:43:03.507889 22841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:03.508108 22841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:03.513634 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.501581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.98.61:39253)
0924 17:43:03.501668 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.501671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.501900 (+ 229us) server_negotiation.cc:407] Connection header received
0924 17:43:03.502026 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.502027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.502037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.502100 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.506987 (+ 4887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.507746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.508769 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.508997 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.509083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.510660 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.510661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.510662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.510669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.512420 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.512822 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.512824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.512824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.512879 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.513091 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.513092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.513092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.513211 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.513329 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.513446 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.513523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 17:43:03.516325 22774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [term 1 FOLLOWER]: Refusing update from remote peer ed74d0a504d04f4f98262647ef8e29ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:03.516656 22821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:03.517978 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed74d0a504d04f4f98262647ef8e29ef. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.518070 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.522827 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.507392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.98.62:34895)
0924 17:43:03.507432 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.507434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.514039 (+ 6605us) server_negotiation.cc:407] Connection header received
0924 17:43:03.514158 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.514159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.514173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.514203 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.514867 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.515622 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.516648 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.516926 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.516981 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.518582 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.518583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.518584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.518591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.520801 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.521565 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.521566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.521567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.521641 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.521990 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.521991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.521992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.522158 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.522267 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.522625 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.522708 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:03.523554 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.507740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.98.60:45885)
0924 17:43:03.507784 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.507787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.515240 (+ 7453us) server_negotiation.cc:407] Connection header received
0924 17:43:03.515297 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.515299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.515311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.515388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.516054 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.516784 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.517898 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.518275 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.518320 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.520157 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.520158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.520159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.520167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.521927 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.522275 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.522277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.522277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.522336 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.522537 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.522538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.522539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.522677 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.522783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.523353 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.523451 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:43:03.524467 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.524573 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:03.524658 22844 mvcc.cc:205] Tried to move back new op lower bound from 7332953225280282624 to 7332953225227091968. Current Snapshot: MvccSnapshot[applied={T|T < 7332953225280282624}]
I20260924 17:43:03.524806 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.524890 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.525180 22841 catalog_manager.cc:1347] Generated new cluster ID: 965ebe400dd848fa8410cc9be9512c7e
I20260924 17:43:03.525228 22841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:03.529343 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } attempt: 3
I20260924 17:43:03.531109 22845 mvcc.cc:205] Tried to move back new op lower bound from 7332953225280282624 to 7332953225227091968. Current Snapshot: MvccSnapshot[applied={T|T < 7332953225280282624}]
Sep 24 17:43:03 dist-test-slave-d0pf krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:03.537959 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.529726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45885 (local address 127.0.0.1:42962)
0924 17:43:03.529778 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:03.529790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:03.529838 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:03.529982 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:03.529983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:03.530177 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:03.530239 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.530241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.530923 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.530924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.531791 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:03.531793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.532811 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.532812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:03.532880 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.533416 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:03.533422 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:03.534974 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:03.536814 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.536815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.536815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.537065 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.537268 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:03.537268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:03.537269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:03.537334 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:03.537633 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:03.537634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:03.537684 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 17:43:03.537757 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:43:03.537863 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 17:43:03.537957 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.529844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.98.60:45885)
0924 17:43:03.529882 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.529884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.529892 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:03.529912 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.529912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.529921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.529976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.530300 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.530843 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.531964 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.532271 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.532333 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.535076 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:03.535078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:03.535078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:03.535085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:03.536745 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.537142 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.537143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.537144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.537201 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:03.537401 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:03.537402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:03.537402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:03.537531 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:03.537604 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.537793 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.537864 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:03.538537 22841 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:03.539428 22841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:03.539917 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: Bootstrap starting.
I20260924 17:43:03.540541 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:03.540848 22725 log.cc:824] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:03.541697 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: No bootstrap required, opened a new log
I20260924 17:43:03.543781 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.543926 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78d1f19b5d944a0ba5c1f8cd5ddf372, State: Initialized, Role: FOLLOWER
I20260924 17:43:03.544011 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } }
I20260924 17:43:03.544381 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:03.544680 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:03.545660 22698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [term 1 FOLLOWER]: Refusing update from remote peer ed74d0a504d04f4f98262647ef8e29ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:03.545686 22699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed74d0a504d04f4f98262647ef8e29ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372"
I20260924 17:43:03.545959 22839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:03.546767 22699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed74d0a504d04f4f98262647ef8e29ef in term 1.
I20260924 17:43:03.549808 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.549885 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.549934 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed74d0a504d04f4f98262647ef8e29ef. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.549986 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.550022 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.550078 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:03.550467 22869 catalog_manager.cc:1259] Loaded cluster ID: 965ebe400dd848fa8410cc9be9512c7e
I20260924 17:43:03.550568 22869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: loading cluster ID for follower catalog manager: success
I20260924 17:43:03.550895 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed74d0a504d04f4f98262647ef8e29ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed74d0a504d04f4f98262647ef8e29ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } peers { permanent_uuid: "a78d1f19b5d944a0ba5c1f8cd5ddf372" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39253 } } peers { permanent_uuid: "3296d6bc9c5f492586665c6d15b5a1e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45885 } } }
I20260924 17:43:03.550997 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:03.551920 22869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: acquiring CA information for follower catalog manager: success
I20260924 17:43:03.552294 22869 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a78d1f19b5d944a0ba5c1f8cd5ddf372: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:43:03.555145 22841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed74d0a504d04f4f98262647ef8e29ef: Generated new TSK 0
I20260924 17:43:03.555346 22841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:43:03.578852 22653 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.577160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.62:34895)
0924 17:43:03.577206 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.577208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.578151 (+ 943us) server_negotiation.cc:407] Connection header received
0924 17:43:03.578259 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.578260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.578272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.578306 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.578777 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32940: BlockingRecv error: recv got EOF from 127.0.0.1:32940 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:43:03.579689 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.577512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.0.98.60:45885)
0924 17:43:03.577552 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.577554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.579163 (+ 1609us) server_negotiation.cc:407] Connection header received
0924 17:43:03.579261 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.579262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.579274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.579303 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.579624 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42974: BlockingRecv error: recv got EOF from 127.0.0.1:42974 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 17:43:03.580402 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.577849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.0.98.61:39253)
0924 17:43:03.577915 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.577918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.579903 (+ 1985us) server_negotiation.cc:407] Connection header received
0924 17:43:03.579999 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.580000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.580011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.580041 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.580330 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41950: BlockingRecv error: recv got EOF from 127.0.0.1:41950 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:43:03.587232 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.583672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.0.98.61:39253)
0924 17:43:03.583708 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.583710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.583958 (+ 248us) server_negotiation.cc:407] Connection header received
0924 17:43:03.584065 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.584066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.584110 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.584158 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.584355 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.585139 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.586259 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.586559 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.586597 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.586818 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.586951 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.587000 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.587134 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.587154 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:43:03.590471 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.589698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.0.98.60:45885)
0924 17:43:03.589736 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.589738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.589926 (+ 188us) server_negotiation.cc:407] Connection header received
0924 17:43:03.590023 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.590024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.590039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.590067 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.590409 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42982: BlockingRecv error: recv got EOF from 127.0.0.1:42982 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:03.595813 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.592755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.98.61:39253)
0924 17:43:03.592789 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.592791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.593004 (+ 213us) server_negotiation.cc:407] Connection header received
0924 17:43:03.593095 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.593096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.593105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.593134 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.593297 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.593972 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.594991 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.595295 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.595337 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.595538 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.595570 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.595598 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.595718 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.595740 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 17:43:03.598989 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.598098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.0.98.60:45885)
0924 17:43:03.598137 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.598139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.598398 (+ 259us) server_negotiation.cc:407] Connection header received
0924 17:43:03.598502 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.598503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.598514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.598544 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.598934 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42994: BlockingRecv error: recv got EOF from 127.0.0.1:42994 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:03.605373 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.602185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.98.61:39253)
0924 17:43:03.602224 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.602226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.602439 (+ 213us) server_negotiation.cc:407] Connection header received
0924 17:43:03.602533 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.602534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.602544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.602574 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.602757 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.603545 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.604581 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.604859 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.604903 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.605099 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.605134 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.605162 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.605268 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.605289 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:43:03.608716 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.607811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.98.60:45885)
0924 17:43:03.607854 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.607856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.608129 (+ 273us) server_negotiation.cc:407] Connection header received
0924 17:43:03.608233 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.608234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.608245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.608275 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.608656 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43008: BlockingRecv error: recv got EOF from 127.0.0.1:43008 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:03.617389 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.614041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.0.98.61:39253)
0924 17:43:03.614085 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.614087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.614337 (+ 250us) server_negotiation.cc:407] Connection header received
0924 17:43:03.614445 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.614446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.614457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.614486 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.614701 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.615451 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.616560 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.616858 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.616904 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.617116 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.617150 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.617178 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.617287 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.617307 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:43:03.620749 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.619844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.0.98.60:45885)
0924 17:43:03.619890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.619892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.620156 (+ 264us) server_negotiation.cc:407] Connection header received
0924 17:43:03.620263 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.620264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.620275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.620306 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.620686 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43024: BlockingRecv error: recv got EOF from 127.0.0.1:43024 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:03.633515 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.630100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.61:39253)
0924 17:43:03.630155 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.630157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.630402 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:43:03.630508 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.630509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.630520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.630548 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.630757 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.631535 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.632696 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.632995 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.633038 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.633244 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.633277 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.633304 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.633414 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.633435 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 17:43:03.636957 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.636133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.60:45885)
0924 17:43:03.636175 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.636177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.636361 (+ 184us) server_negotiation.cc:407] Connection header received
0924 17:43:03.636462 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.636463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.636475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.636506 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.636894 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43036: BlockingRecv error: recv got EOF from 127.0.0.1:43036 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:03.657701 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.654350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.98.61:39253)
0924 17:43:03.654395 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.654397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.654665 (+ 268us) server_negotiation.cc:407] Connection header received
0924 17:43:03.654768 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.654769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.654780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.654809 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.654997 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.655764 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.656884 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.657159 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.657204 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.657412 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.657446 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.657477 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.657583 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.657618 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:43:03.660954 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.660076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.0.98.60:45885)
0924 17:43:03.660121 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.660123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.660381 (+ 258us) server_negotiation.cc:407] Connection header received
0924 17:43:03.660481 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.660482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.660494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.660524 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.660900 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43040: BlockingRecv error: recv got EOF from 127.0.0.1:43040 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:03.696955 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.694333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.98.61:39253)
0924 17:43:03.694383 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.694385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.694548 (+ 163us) server_negotiation.cc:407] Connection header received
0924 17:43:03.694668 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.694669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.694681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.694752 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.694887 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.695461 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.696242 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.696454 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.696531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.696686 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.696726 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.696785 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.696847 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.696865 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 17:43:03.699512 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.698600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.98.60:45885)
0924 17:43:03.698649 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.698651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.698904 (+ 253us) server_negotiation.cc:407] Connection header received
0924 17:43:03.698992 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.698993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.699006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.699112 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.699458 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43042: BlockingRecv error: recv got EOF from 127.0.0.1:43042 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:03.766985 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.764770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.0.98.61:39253)
0924 17:43:03.764837 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.764839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.764940 (+ 101us) server_negotiation.cc:407] Connection header received
0924 17:43:03.765002 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.765003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.765012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.765062 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.765165 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.765635 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.766346 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.766527 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.766579 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.766741 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.766767 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.766804 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.766859 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.766874 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 17:43:03.769381 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.768795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.0.98.60:45885)
0924 17:43:03.768851 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.768853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.768877 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:43:03.768939 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.768940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.768952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.769006 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.769313 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43054: BlockingRecv error: recv got EOF from 127.0.0.1:43054 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:43:03.900970 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:03.898526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.0.98.61:39253)
0924 17:43:03.898597 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.898606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.898655 (+ 49us) server_negotiation.cc:407] Connection header received
0924 17:43:03.898730 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.898731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.898743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.898875 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:03.899011 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.899570 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.900377 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:03.900574 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:03.900639 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:03.900766 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:03.900791 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:03.900833 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:03.900889 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 17:43:03.900903 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 17:43:03.903183 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:03.902591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.98.60:45885)
0924 17:43:03.902636 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:03.902638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:03.902667 (+ 29us) server_negotiation.cc:407] Connection header received
0924 17:43:03.902731 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:03.902732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:03.902742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:03.902787 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:03.903106 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43056: BlockingRecv error: recv got EOF from 127.0.0.1:43056 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:04.162729 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.160360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.0.98.61:39253)
0924 17:43:04.160449 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.160452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.160464 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:43:04.160495 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.160497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.160508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.160585 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.160739 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.161232 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.162020 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.162217 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.162309 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.162457 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.162497 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.162567 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.162612 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.162632 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 17:43:04.165261 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:04.164716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.0.98.60:45885)
0924 17:43:04.164785 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.164787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.164796 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:04.164830 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.164831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.164841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.164886 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:04.165200 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43072: BlockingRecv error: recv got EOF from 127.0.0.1:43072 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:43:04.496093 22818 catalog_manager.cc:1259] Loaded cluster ID: 965ebe400dd848fa8410cc9be9512c7e
I20260924 17:43:04.496196 22818 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: loading cluster ID for follower catalog manager: success
I20260924 17:43:04.497001 22818 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: acquiring CA information for follower catalog manager: success
I20260924 17:43:04.497321 22818 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3296d6bc9c5f492586665c6d15b5a1e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:43:04.681141 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.678441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.0.98.61:39253)
0924 17:43:04.678719 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.678721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.678731 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:04.678758 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.678758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.678770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.678831 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.679007 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.679620 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.680442 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.680658 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.680718 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.680861 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.680894 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.680949 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.680990 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.681013 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":86,"threads_started":1}
I20260924 17:43:04.685456 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.682937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.0.98.60:45885)
0924 17:43:04.683081 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.683083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.683092 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:04.683160 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.683161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.683219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.683302 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.683443 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.683924 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.684652 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.684859 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.684942 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.685120 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.685226 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.685310 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.685337 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.685358 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":53,"threads_started":1}
I20260924 17:43:04.690160 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.687346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.0.98.60:45885)
0924 17:43:04.687382 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.687383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.687496 (+ 113us) server_negotiation.cc:407] Connection header received
0924 17:43:04.687601 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.687602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.687610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.687663 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.687811 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.688508 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689283 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.689540 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689602 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.689740 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.689770 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.689835 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.690050 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.690078 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:04.690191 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.687116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.98.61:39253)
0924 17:43:04.687162 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.687164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.687463 (+ 299us) server_negotiation.cc:407] Connection header received
0924 17:43:04.687553 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.687554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.687567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.687597 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.687780 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.688323 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689476 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.689766 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689833 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.689955 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.689997 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.690047 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.690089 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.690112 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:04.690529 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:04.687144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.98.62:34895)
0924 17:43:04.687293 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:04.687295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:04.687305 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:04.687379 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:04.687381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:04.687434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:04.687522 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:04.687672 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.688460 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689487 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:04.689798 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:04.689853 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:04.690070 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:04.690179 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:04.690240 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:04.690419 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 17:43:04.690445 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":50,"threads_started":1}
I20260924 17:43:04.692039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22582
I20260924 17:43:04.697686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22657
I20260924 17:43:04.702505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 22732
2026-09-24T17:43:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2833 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:04 dist-test-slave-d0pf krb5kdc[23002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:04 dist-test-slave-d0pf krb5kdc[23002](info): set up 2 sockets
Sep 24 17:43:04 dist-test-slave-d0pf krb5kdc[23002](info): commencing operation
krb5kdc: starting...
W20260924 17:43:05.747313 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/test-user.keytab.
Sep 24 17:43:05 dist-test-slave-d0pf krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 17:43:05.866037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33731
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:05.947302 23015 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:05.947536 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:05.947554 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:05.947575 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:05.948767 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:05.948810 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:05.948824 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:05.948836 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:05.948849 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:05.950433 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:05.950640 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:05.950815 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:05.955670 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:05.955735 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:05.955744 23015 server_base.cc:1034] running on GCE node
W20260924 17:43:05.955662 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:05.956174 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:05.956514 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:05.957688 23015 hybrid_clock.cc:648] HybridClock initialized: now 1790271785957667 us; error 42 us; skew 500 ppm
Sep 24 17:43:05 dist-test-slave-d0pf krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:05.960291 23015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 17:43:05.960788 23015 webserver.cc:460] Webserver started at http://127.0.98.62:43133/ using document root <none> and password file <none>
I20260924 17:43:05.961018 23015 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:05.961087 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:05.961239 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:05.962317 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data/instance:
uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932"
format_stamp: "Formatted at 2026-09-24 17:43:05 on dist-test-slave-d0pf"
server_key: "b660fb032045b31f903a5ce6db2e1959"
server_key_iv: "1547444096622df91d0438f70d1cb2f6"
server_key_version: "encryptionkey@0"
I20260924 17:43:05.962697 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932"
format_stamp: "Formatted at 2026-09-24 17:43:05 on dist-test-slave-d0pf"
server_key: "b660fb032045b31f903a5ce6db2e1959"
server_key_iv: "1547444096622df91d0438f70d1cb2f6"
server_key_version: "encryptionkey@0"
I20260924 17:43:05.964198 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 17:43:05.965087 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:05.965327 23015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:05.965449 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal
uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932"
format_stamp: "Formatted at 2026-09-24 17:43:05 on dist-test-slave-d0pf"
server_key: "b660fb032045b31f903a5ce6db2e1959"
server_key_iv: "1547444096622df91d0438f70d1cb2f6"
server_key_version: "encryptionkey@0"
I20260924 17:43:05.965526 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:05.976696 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:05.979153 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:05.979324 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:05.983417 23015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33731
I20260924 17:43:05.983449 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33731 every 8 connection(s)
I20260924 17:43:05.983819 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:05.984773 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:05.986330 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:05.990895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 23015
I20260924 17:43:05.991144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-0/wal/instance
I20260924 17:43:05.991353 392 external_mini_cluster.cc:1428] Setting key 9c4ad1290a6f9935ba1076ccf1043373
Sep 24 17:43:05 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:05.996443 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:05.986986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.0.1:33708)
0924 17:43:05.987164 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:05.987203 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:05.987656 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:05.987947 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:05.987949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:05.988187 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:05.988336 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:05.988341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:05.989320 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:05.989321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:05.989992 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:05.989994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:05.990483 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:05.990484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:05.990536 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:05.991139 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:05.991150 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:05.992802 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:05.994891 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:05.994896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:05.994898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:05.995177 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:05.995439 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:05.995440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:05.995440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:05.995516 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:05.996015 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:05.996022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:05.996081 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 17:43:05.996156 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 17:43:05.996285 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260924 17:43:05.996562 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:05.987073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.62:33731)
0924 17:43:05.987203 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:05.987211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:05.987281 (+ 70us) server_negotiation.cc:407] Connection header received
0924 17:43:05.987781 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:05.987783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:05.987833 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:05.987915 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:05.988414 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:05.989218 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:05.990082 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:05.990385 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:05.990474 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:05.992886 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:05.992894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:05.992895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:05.992903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:05.994799 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:05.995255 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:05.995259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:05.995259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:05.995367 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:05.995591 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:05.995592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:05.995593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:05.995917 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:05.996017 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:05.996235 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 17:43:05.996377 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260924 17:43:05.997435 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:05.997817 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:05.997676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33078)
0924 17:43:05.997726 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:05.997797 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:43:05.997988 23033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
I20260924 17:43:05.998412 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:05.998323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33088)
0924 17:43:05.998373 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:05.998397 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 17:43:05.998558 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41555: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 17:43:06.017226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41555
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:06.047813 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } attempt: 1
I20260924 17:43:06.048323 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.048187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33090)
0924 17:43:06.048251 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.048293 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:43:06.048702 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.048608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33096)
0924 17:43:06.048655 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.048678 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:43:06.048806 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41555: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
W20260924 17:43:06.097019 23090 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:06.097280 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:06.097307 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:06.097338 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:06.098506 23090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:06.098573 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:06.098639 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:06.098672 23090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:06.098697 23090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:06.100345 23090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:06.100626 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:06.100883 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:06.105643 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:06.105662 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:06.105640 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:06.105933 23090 server_base.cc:1034] running on GCE node
I20260924 17:43:06.106166 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:06.106462 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:06.107610 23090 hybrid_clock.cc:648] HybridClock initialized: now 1790271786107594 us; error 25 us; skew 500 ppm
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:06.109995 23090 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 17:43:06.110419 23090 webserver.cc:460] Webserver started at http://127.0.98.61:45167/ using document root <none> and password file <none>
I20260924 17:43:06.110659 23090 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:06.110725 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:06.110874 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:06.112026 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data/instance:
uuid: "f7643f350bf742ca8e3802908e81cf80"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "914ca41a0223fbe76bba87ce378bcfc2"
server_key_iv: "9681d81fee69f99b392e4516ef68380c"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.112383 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal/instance:
uuid: "f7643f350bf742ca8e3802908e81cf80"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "914ca41a0223fbe76bba87ce378bcfc2"
server_key_iv: "9681d81fee69f99b392e4516ef68380c"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.113679 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:06.114365 23105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:06.114521 23090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:06.114614 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal
uuid: "f7643f350bf742ca8e3802908e81cf80"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "914ca41a0223fbe76bba87ce378bcfc2"
server_key_iv: "9681d81fee69f99b392e4516ef68380c"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.114691 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:06.116963 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } attempt: 2
I20260924 17:43:06.117390 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.117256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33100)
0924 17:43:06.117317 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.117355 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:43:06.117764 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.117655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33116)
0924 17:43:06.117703 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.117729 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:43:06.117879 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41555: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41555: connect: Connection refused (error 111)
I20260924 17:43:06.128733 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:06.130843 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:06.131001 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:06.134956 23090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41555
I20260924 17:43:06.135068 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41555 every 8 connection(s)
I20260924 17:43:06.135392 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/data/info.pb
I20260924 17:43:06.136384 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:06.137837 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:06.141929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 23090
I20260924 17:43:06.142040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-1/wal/instance
I20260924 17:43:06.142247 392 external_mini_cluster.cc:1428] Setting key bb668e302809d1cd4190ade41da1e5e8
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:06.146886 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.138542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.62:33731)
0924 17:43:06.138602 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.138604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.138787 (+ 183us) server_negotiation.cc:407] Connection header received
0924 17:43:06.139269 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.139270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.139279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.139344 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.139907 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.140374 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.141298 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.141481 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.141530 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.143761 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.143763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.143764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.143772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.145475 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.145859 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.145861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.145861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.145928 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.146186 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.146187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.146188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.146346 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.146418 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.146661 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.146752 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:43:06.146907 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.138508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.0.1:33712)
0924 17:43:06.138658 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.138698 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.139189 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.139397 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.139400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.139638 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.139827 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.139830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.140451 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.140452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.141227 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.141228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.141559 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.141560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.141666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.142221 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.142228 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.143665 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.145558 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.145564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.145565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.145779 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.146011 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.146011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.146012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.146118 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.146412 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.146417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.146531 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.146613 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.146730 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260924 17:43:06.147684 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:06.156673 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.147930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33132)
0924 17:43:06.147977 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.147987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.148027 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.148282 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.148283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.148419 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.148463 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.148465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.149304 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.149305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.149783 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.149785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.150359 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.150360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.150422 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.150797 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.150803 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.153512 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.155540 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.155541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.155541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.155703 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.155968 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.155969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.155970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.156026 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.156312 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.156313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.156352 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.156448 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.156551 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 17:43:06.156773 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.148001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.98.61:41555)
0924 17:43:06.148156 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.148160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.148176 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:43:06.148192 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.148194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.148203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.148282 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.148543 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.149200 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.149884 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.150258 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.150363 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.153606 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.153614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.153614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.153620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.155449 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.155834 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.155837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.155838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.155898 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.156091 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.156092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.156092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.156241 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.156333 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.156551 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.156613 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260924 17:43:06.157778 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:06.158159 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.158027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54740)
0924 17:43:06.158078 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.158126 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:43:06.158303 23106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
I20260924 17:43:06.158689 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.158594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54748)
0924 17:43:06.158644 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.158667 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 17:43:06.158823 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41281: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 17:43:06.169194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcbbHA6/build/release/bin/kudu
/tmp/dist-test-taskcbbHA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41281
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:06.208035 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } attempt: 1
I20260924 17:43:06.208523 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.208387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54760)
0924 17:43:06.208450 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.208493 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:43:06.208866 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.208766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54762)
0924 17:43:06.208810 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.208839 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:43:06.208963 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41281: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
I20260924 17:43:06.209000 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } attempt: 3
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:06.216418 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.209307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33140)
0924 17:43:06.209363 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.209374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.209416 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.209557 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.209558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.209740 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.209815 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.209818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.210470 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.210471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.211330 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.211333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.211712 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.211713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.211806 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.212253 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.212258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.213606 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.215351 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.215352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.215353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.215531 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.215760 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.215761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.215761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.215828 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.216082 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.216083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.216130 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.216209 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.216322 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 17:43:06.216416 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.209358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.0.98.61:41555)
0924 17:43:06.209392 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.209394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.209405 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:06.209471 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.209471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.209480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.209531 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.209888 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.210377 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.211409 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.211622 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.211678 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.213710 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.213711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.213712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.213719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.215233 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.215619 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.215620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.215621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.215678 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.215892 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.215893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.215893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.216009 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.216080 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.216246 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.216322 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:06.216894 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:06.217255 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.217156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54770)
0924 17:43:06.217203 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.217236 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 17:43:06.217607 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.217499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54776)
0924 17:43:06.217550 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.217579 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:43:06.217713 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41281: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
I20260924 17:43:06.248921 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } attempt: 1
I20260924 17:43:06.249457 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.249305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54786)
0924 17:43:06.249370 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.249419 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:43:06.249842 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.249746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54802)
0924 17:43:06.249791 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.249820 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:43:06.249945 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41281: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
W20260924 17:43:06.252089 23165 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:43:06.252281 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:06.252298 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:06.252318 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:06.253379 23165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:06.253412 23165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:06.253427 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:06.253439 23165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:06.253451 23165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:06.254848 23165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33731,127.0.98.61:41555,127.0.98.60:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:06.255096 23165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:06.255304 23165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:06.260267 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:06.260227 23173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:06.260246 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:06.260476 23165 server_base.cc:1034] running on GCE node
I20260924 17:43:06.260747 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:06.261098 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:06.262269 23165 hybrid_clock.cc:648] HybridClock initialized: now 1790271786262262 us; error 37 us; skew 500 ppm
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:06.264914 23165 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 17:43:06.265403 23165 webserver.cc:460] Webserver started at http://127.0.98.60:33555/ using document root <none> and password file <none>
I20260924 17:43:06.265648 23165 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:06.265720 23165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:06.265872 23165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:06.266927 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data/instance:
uuid: "e7253d6617144b58a385ebfe2e8f5c8a"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "5f52ab81189d73a838f5b5eea9db9a8c"
server_key_iv: "b7b2034d767447482a0699cfaea198ff"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.267342 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal/instance:
uuid: "e7253d6617144b58a385ebfe2e8f5c8a"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "5f52ab81189d73a838f5b5eea9db9a8c"
server_key_iv: "b7b2034d767447482a0699cfaea198ff"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.268708 23165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:06.269472 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:06.269632 23165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:06.269737 23165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal
uuid: "e7253d6617144b58a385ebfe2e8f5c8a"
format_stamp: "Formatted at 2026-09-24 17:43:06 on dist-test-slave-d0pf"
server_key: "5f52ab81189d73a838f5b5eea9db9a8c"
server_key_iv: "b7b2034d767447482a0699cfaea198ff"
server_key_version: "encryptionkey@0"
I20260924 17:43:06.269829 23165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:06.277125 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } attempt: 2
I20260924 17:43:06.277555 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.277424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54808)
0924 17:43:06.277485 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.277525 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:43:06.277887 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.277796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54816)
0924 17:43:06.277839 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.277863 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 17:43:06.277966 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41281: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41281: connect: Connection refused (error 111)
I20260924 17:43:06.282883 23165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:06.284855 23165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:06.285012 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:06.289239 23165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41281
I20260924 17:43:06.289292 23232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41281 every 8 connection(s)
I20260924 17:43:06.289672 23165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/data/info.pb
I20260924 17:43:06.290712 23233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:06.292371 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:06.293939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcbbHA6/build/release/bin/kudu as pid 23165
I20260924 17:43:06.294060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcbbHA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271516406493-392-0/minicluster-data/master-2/wal/instance
I20260924 17:43:06.294268 392 external_mini_cluster.cc:1428] Setting key 757881ab32b7598212df9fc483f1b0a6
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:06.303339 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.293101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.98.62:33731)
0924 17:43:06.293166 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.293168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.293375 (+ 207us) server_negotiation.cc:407] Connection header received
0924 17:43:06.293899 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.293901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.293914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.293965 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.294736 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.295565 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.296962 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.297258 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.297339 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.299738 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.299739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.299740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.299783 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.301910 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.302332 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.302333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.302333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.302382 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.302651 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.302652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.302652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.302795 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.302869 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.303168 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.303260 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:43:06.303550 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.293071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.0.1:33728)
0924 17:43:06.293256 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.293294 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.293766 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.294054 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.294059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.294395 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.294621 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.294627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.295660 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.295661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.296840 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.296842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.297360 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.297361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.297456 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.298012 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.298020 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.299635 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.301996 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.302001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.302003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.302253 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.302448 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.302449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.302449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.302577 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.302863 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.302867 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.303020 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.303146 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.303314 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260924 17:43:06.303902 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.294646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.98.60:41281)
0924 17:43:06.294795 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.294799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.294926 (+ 127us) server_negotiation.cc:407] Connection header received
0924 17:43:06.294946 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.294947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.294963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.295023 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.295785 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.296900 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.297818 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.298121 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.298201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.300890 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.300897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.300897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.300903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.302585 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.302963 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.302965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.302965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.303019 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.303228 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.303229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.303229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.303399 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.303533 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.303719 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.303802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20260924 17:43:06.304301 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 17:43:06.311519 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.304558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41555 (local address 127.0.0.1:33154)
0924 17:43:06.304661 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.304673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.304757 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.304911 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.304912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.305066 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.305127 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.305129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.305949 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.305950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.306703 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.306732 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.307125 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.307126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.307185 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.307544 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.307549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.308814 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.310499 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.310500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.310501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.310743 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.310947 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.310948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.310948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.311014 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.311272 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.311273 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.311323 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.311394 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.311458 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 17:43:06.311625 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.304615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.98.61:41555)
0924 17:43:06.304682 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.304684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.304732 (+ 48us) server_negotiation.cc:407] Connection header received
0924 17:43:06.304816 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.304817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.304828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.304907 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.305204 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.305857 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.306814 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.307069 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.307129 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.308898 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.308899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.308900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.308907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.310409 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.310827 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.310828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.310828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.310879 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.311101 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.311102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.311102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.311201 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.311251 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.311430 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.311505 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:43:06.311985 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:43:06.312498 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.304651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.98.62:33731)
0924 17:43:06.304696 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.304697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.304790 (+ 93us) server_negotiation.cc:407] Connection header received
0924 17:43:06.304831 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.304832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.304841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.304906 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.305435 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.306071 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.306976 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.307257 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.307340 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.309541 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.309542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.309542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.309549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.311224 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.311628 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.311629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.311630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.311702 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.311917 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.311918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.311918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.312045 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.312219 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.312301 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.312390 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 17:43:06.319271 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.312236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54830)
0924 17:43:06.312292 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.312304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.312342 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.312532 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.312533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.312679 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.312716 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.312723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.313657 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.313657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.314138 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.314140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.314532 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.314533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.314579 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.314952 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.314956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.316099 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.317887 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.317888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.317888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.318135 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.318366 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.318367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.318367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.318435 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.318746 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.318747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.318901 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.318993 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.319130 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 17:43:06.319271 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.312306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.0.98.60:41281)
0924 17:43:06.312449 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.312451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.312457 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:43:06.312472 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.312473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.312481 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.312523 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.312810 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.313582 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.314235 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.314448 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.314532 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.316193 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.316195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.316195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.316202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.317813 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.318207 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.318208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.318208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.318269 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.318524 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.318525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.318525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.318664 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.318731 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.319023 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.319190 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":127}
I20260924 17:43:06.320696 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.313136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.98.61:41555)
0924 17:43:06.313170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.313172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.313182 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:06.313201 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.313202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.313213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.313273 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.313822 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.314257 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.315063 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.315291 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.315374 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.317889 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.317891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.317891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.317898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.319677 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.320006 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.320007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.320008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.320067 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.320212 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.320213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.320213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.320343 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.320498 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.320534 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.320606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:43:06.321072 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: Bootstrap starting.
I20260924 17:43:06.322028 23233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:06.322037 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:06.322312 23233 log.cc:824] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:06.323021 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: No bootstrap required, opened a new log
I20260924 17:43:06.322916 23057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33730
I20260924 17:43:06.324481 23233 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.324599 23233 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7253d6617144b58a385ebfe2e8f5c8a, State: Initialized, Role: FOLLOWER
I20260924 17:43:06.324687 23233 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.325012 23233 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:06.325014 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.325066 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } attempt: 2
I20260924 17:43:06.325119 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:43:06.327143 23250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:06.327198 23250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:06.327667 23165 master_runner.cc:186] Error getting master registration for 127.0.98.62:33731: OK, instance_id { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" instance_seqno: 1790271785982207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:43:06.330670 23165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:43:06.333050 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.325273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54832)
0924 17:43:06.325338 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.325350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.325397 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.325535 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.325536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.325721 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.325768 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.325770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.326687 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.326688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.327529 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.327532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.328024 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.328025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.328079 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.328493 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.328499 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.330042 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.331946 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.331947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.331947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.332167 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.332380 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.332380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.332381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.332441 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.332735 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.332736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.332785 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.332853 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.332946 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 17:43:06.333055 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.325324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.0.98.60:41281)
0924 17:43:06.325361 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.325363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.325386 (+ 23us) server_negotiation.cc:407] Connection header received
0924 17:43:06.325453 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.325453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.325461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.325530 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.325858 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.326588 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.327626 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.327911 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.327976 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.330143 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.330145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.330145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.330152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.331832 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.332247 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.332248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.332248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.332309 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.332508 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.332509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.332509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.332664 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.332738 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.332892 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.332969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:06.334560 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: Bootstrap starting.
I20260924 17:43:06.335325 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:06.335631 23083 log.cc:824] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:06.336549 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: No bootstrap required, opened a new log
I20260924 17:43:06.337914 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.338058 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9fc8ba19ea4b3baa0ceaecc7182932, State: Initialized, Role: FOLLOWER
I20260924 17:43:06.338140 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.338465 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:06.338490 23057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 17:43:06.338584 23057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:06.339306 23057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.339576 23057 leader_election.cc:288] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [CANDIDATE]: Term 1 election: Requested vote from peers f7643f350bf742ca8e3802908e81cf80 (127.0.98.61:41555), e7253d6617144b58a385ebfe2e8f5c8a (127.0.98.60:41281)
W20260924 17:43:06.340620 23264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:06.340670 23264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:06.340705 23207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e7253d6617144b58a385ebfe2e8f5c8a"
I20260924 17:43:06.340852 23207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:06.341790 23207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9fc8ba19ea4b3baa0ceaecc7182932 in term 1.
I20260924 17:43:06.342025 23031 leader_election.cc:302] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f9fc8ba19ea4b3baa0ceaecc7182932, e7253d6617144b58a385ebfe2e8f5c8a; no voters:
I20260924 17:43:06.338438 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.342319 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.342374 23253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:06.342423 23253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [term 1 LEADER]: Becoming Leader. State: Replica: 4f9fc8ba19ea4b3baa0ceaecc7182932, State: Running, Role: LEADER
I20260924 17:43:06.342504 23253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.342941 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f9fc8ba19ea4b3baa0ceaecc7182932. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.343024 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:06.343621 23272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:06.341289 23015 master_runner.cc:186] Error getting master registration for 127.0.98.61:41555: OK, instance_id { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" instance_seqno: 1790271786133901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:43:06.343829 23015 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:43:06.343829 23272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:06.353031 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.343358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.0.98.60:41281)
0924 17:43:06.343399 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.343401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.345517 (+ 2116us) server_negotiation.cc:407] Connection header received
0924 17:43:06.345594 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.345595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.345606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.345636 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.346441 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.346911 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.347969 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.348263 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.348325 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.349948 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.349949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.349950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.349956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.351708 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.352142 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.352144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.352144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.352200 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.352432 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.352433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.352433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.352560 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.352650 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.352864 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.352940 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:06.353447 23207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [term 1 FOLLOWER]: Refusing update from remote peer 4f9fc8ba19ea4b3baa0ceaecc7182932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:06.353839 23253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:06.355162 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f9fc8ba19ea4b3baa0ceaecc7182932. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.355252 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.361821 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.345256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.98.62:33731)
0924 17:43:06.345301 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.345303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.353799 (+ 8496us) server_negotiation.cc:407] Connection header received
0924 17:43:06.353896 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.353897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.353908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.353938 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.354588 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.355284 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.356284 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.356564 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.356603 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.358233 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.358235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.358235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.358243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.359987 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.360605 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.360607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.360607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.360672 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.361022 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.361023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.361024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.361163 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.361258 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.361621 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.361697 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:06.362344 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.362454 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:06.362728 23272 catalog_manager.cc:1347] Generated new cluster ID: ce1a4a539683420ab804ad7aea958469
I20260924 17:43:06.362780 23272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:06.363087 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.363149 23274 mvcc.cc:205] Tried to move back new op lower bound from 7332953236895555584 to 7332953236860096512. Current Snapshot: MvccSnapshot[applied={T|T < 7332953236895555584 or (T in {7332953236895555584})}]
I20260924 17:43:06.363188 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.363674 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.345081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.98.61:41555)
0924 17:43:06.345138 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.345140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.351693 (+ 6553us) server_negotiation.cc:407] Connection header received
0924 17:43:06.351806 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.351807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.351820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.351851 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.352515 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.353188 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.356073 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.356378 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.356437 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.359584 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.359585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.359586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.359594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.361413 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.362235 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.362248 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.362249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.362321 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.362722 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.362723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.362723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.362881 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.362963 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.363486 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.363573 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:06.369055 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } attempt: 3
Sep 24 17:43:06 dist-test-slave-d0pf krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 17:43:06.375406 23272 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:06.376121 23272 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:06.377722 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.369355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41281 (local address 127.0.0.1:54844)
0924 17:43:06.369407 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:06.369417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:06.369453 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:06.369593 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:06.369593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:06.369836 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:06.369884 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.369886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.370904 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.370905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.371748 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:06.371751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.372281 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.372282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:06.372356 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.372821 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:06.372826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:06.374381 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:06.376511 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.376512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.376513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.376738 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.377008 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:06.377009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:06.377009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:06.377079 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:06.377400 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:06.377401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:06.377458 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 17:43:06.377520 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:43:06.377619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:43:06.378284 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.369430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.98.60:41281)
0924 17:43:06.369487 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.369489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.369497 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:06.369520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.369521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.369531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.369577 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.370074 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.370830 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.371860 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.372144 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.372203 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.374474 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:06.374475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:06.374476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:06.374483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:06.376362 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.376843 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.376844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.376844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.376927 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:06.377154 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:06.377155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:06.377155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:06.377298 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:06.377358 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.378113 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.378195 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:43:06.380136 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: Bootstrap starting.
I20260924 17:43:06.380873 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:06.381187 23158 log.cc:824] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:06.388625 23272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932: Generated new TSK 0
I20260924 17:43:06.389014 23272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:06.389617 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: No bootstrap required, opened a new log
I20260924 17:43:06.391784 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.391937 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7643f350bf742ca8e3802908e81cf80, State: Initialized, Role: FOLLOWER
I20260924 17:43:06.392014 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } }
I20260924 17:43:06.392349 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:06.395848 23131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:06.396865 23131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [term 1 FOLLOWER]: Refusing update from remote peer 4f9fc8ba19ea4b3baa0ceaecc7182932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:06.397126 23132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7643f350bf742ca8e3802908e81cf80"
I20260924 17:43:06.397192 23253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:06.398187 23132 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9fc8ba19ea4b3baa0ceaecc7182932 in term 1.
I20260924 17:43:06.400604 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.400708 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.401160 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f9fc8ba19ea4b3baa0ceaecc7182932. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.401293 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.401324 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:06.401360 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:06.402918 23295 mvcc.cc:205] Tried to move back new op lower bound from 7332953237043863552 to 7332953236860096512. Current Snapshot: MvccSnapshot[applied={T|T < 7332953236895555584}]
I20260924 17:43:06.405187 23293 catalog_manager.cc:1259] Loaded cluster ID: ce1a4a539683420ab804ad7aea958469
I20260924 17:43:06.405331 23293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: loading cluster ID for follower catalog manager: success
I20260924 17:43:06.406554 23293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: acquiring CA information for follower catalog manager: success
I20260924 17:43:06.406953 23293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7643f350bf742ca8e3802908e81cf80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 17:43:06.423882 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.421415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.0.98.60:41281)
0924 17:43:06.421460 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.421463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.422356 (+ 893us) server_negotiation.cc:407] Connection header received
0924 17:43:06.422686 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.422688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.422701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.422731 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.423807 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54858: BlockingRecv error: recv got EOF from 127.0.0.1:54858 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:43:06.424295 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.421660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.98.61:41555)
0924 17:43:06.421697 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.421698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.422447 (+ 749us) server_negotiation.cc:407] Connection header received
0924 17:43:06.422813 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.422815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.422825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.422854 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.424226 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33192: BlockingRecv error: recv got EOF from 127.0.0.1:33192 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 17:43:06.424697 23086 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.421867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.0.98.62:33731)
0924 17:43:06.421905 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.421907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.422634 (+ 727us) server_negotiation.cc:407] Connection header received
0924 17:43:06.423317 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.423319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.423329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.423358 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.424638 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33756: BlockingRecv error: recv got EOF from 127.0.0.1:33756 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:06.429780 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.426266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.0.98.61:41555)
0924 17:43:06.426299 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.426301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.426510 (+ 209us) server_negotiation.cc:407] Connection header received
0924 17:43:06.426614 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.426614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.426652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.426697 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.426908 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.427706 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.428852 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.429136 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.429175 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.429391 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.429520 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.429569 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.429677 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.429695 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 17:43:06.436157 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.435311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.0.98.60:41281)
0924 17:43:06.435354 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.435356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.435601 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:43:06.435710 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.435711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.435730 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.435772 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.436107 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54868: BlockingRecv error: recv got EOF from 127.0.0.1:54868 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:06.441867 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.438400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.0.98.61:41555)
0924 17:43:06.438436 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.438438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.438697 (+ 259us) server_negotiation.cc:407] Connection header received
0924 17:43:06.438811 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.438812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.438821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.438851 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.439122 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.439884 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.441057 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.441326 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.441372 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.441574 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.441609 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.441639 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.441768 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.441791 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:43:06.445217 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.444397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.0.98.60:41281)
0924 17:43:06.444438 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.444440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.444679 (+ 239us) server_negotiation.cc:407] Connection header received
0924 17:43:06.444792 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.444793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.444804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.444832 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.445164 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54874: BlockingRecv error: recv got EOF from 127.0.0.1:54874 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:06.451536 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.448346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.98.61:41555)
0924 17:43:06.448382 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.448383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.448632 (+ 249us) server_negotiation.cc:407] Connection header received
0924 17:43:06.448731 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.448732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.448754 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.448783 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.448964 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.449664 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.450725 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.450997 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.451069 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.451270 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.451304 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.451333 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.451442 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.451461 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 17:43:06.454734 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.453896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.98.60:41281)
0924 17:43:06.453937 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.453939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.454183 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:43:06.454285 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.454287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.454298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.454328 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.454682 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54878: BlockingRecv error: recv got EOF from 127.0.0.1:54878 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:06.463241 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.460003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.0.98.61:41555)
0924 17:43:06.460041 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.460043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.460291 (+ 248us) server_negotiation.cc:407] Connection header received
0924 17:43:06.460391 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.460392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.460401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.460429 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.460607 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.461303 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.462392 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.462667 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.462712 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.462938 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.462972 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.463001 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.463138 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.463158 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 17:43:06.466427 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.465638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.0.98.60:41281)
0924 17:43:06.465681 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.465683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.465884 (+ 201us) server_negotiation.cc:407] Connection header received
0924 17:43:06.465982 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.465983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.465993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.466023 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.466373 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54890: BlockingRecv error: recv got EOF from 127.0.0.1:54890 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:06.479017 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.475701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.0.98.61:41555)
0924 17:43:06.475767 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.475769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.476045 (+ 276us) server_negotiation.cc:407] Connection header received
0924 17:43:06.476144 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.476145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.476155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.476185 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.476374 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.477125 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.478201 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.478475 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.478522 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.478727 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.478775 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.478805 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.478916 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.478937 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 17:43:06.482268 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.481471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.0.98.60:41281)
0924 17:43:06.481512 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.481514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.481696 (+ 182us) server_negotiation.cc:407] Connection header received
0924 17:43:06.481809 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.481810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.481822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.481851 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.482211 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54902: BlockingRecv error: recv got EOF from 127.0.0.1:54902 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:06.503103 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.499655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.0.98.61:41555)
0924 17:43:06.499696 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.499698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.499982 (+ 284us) server_negotiation.cc:407] Connection header received
0924 17:43:06.500079 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.500081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.500091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.500122 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.500323 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.501071 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.502231 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.502516 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.502562 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.502785 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.502817 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.502845 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.502952 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.502972 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 17:43:06.506392 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.505541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.0.98.60:41281)
0924 17:43:06.505587 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.505589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.505800 (+ 211us) server_negotiation.cc:407] Connection header received
0924 17:43:06.505907 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.505909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.505921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.505952 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.506331 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54912: BlockingRecv error: recv got EOF from 127.0.0.1:54912 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:06.542330 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.539869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.61:41555)
0924 17:43:06.539920 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.539922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.540059 (+ 137us) server_negotiation.cc:407] Connection header received
0924 17:43:06.540134 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.540135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.540185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.540233 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.540393 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.540872 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.541632 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.541826 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.541897 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.542051 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.542089 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.542158 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.542216 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.542236 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 17:43:06.544934 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.544340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.0.98.60:41281)
0924 17:43:06.544396 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.544398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.544408 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:06.544480 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.544481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.544492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.544548 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.544871 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54924: BlockingRecv error: recv got EOF from 127.0.0.1:54924 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:43:06.612543 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.610206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.98.61:41555)
0924 17:43:06.610294 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.610296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.610310 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:43:06.610399 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.610400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.610409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.610462 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.610598 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.611111 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.611912 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.612092 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.612149 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.612287 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.612313 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.612352 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.612415 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.612430 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 17:43:06.614820 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.614265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.0.98.60:41281)
0924 17:43:06.614339 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.614341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.614352 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:06.614397 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.614398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.614409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.614465 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.614764 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54932: BlockingRecv error: recv got EOF from 127.0.0.1:54932 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:43:06.746217 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:06.743895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.61:41555)
0924 17:43:06.743975 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.743977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.744051 (+ 74us) server_negotiation.cc:407] Connection header received
0924 17:43:06.744100 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.744101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.744110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.744164 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:06.744316 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.744809 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.745540 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:06.745742 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:06.745801 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:06.745950 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:06.745982 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:06.746036 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:06.746080 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 17:43:06.746096 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 17:43:06.748567 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:06.747963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.0.98.60:41281)
0924 17:43:06.748023 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:06.748027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:06.748055 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:43:06.748108 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:06.748109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:06.748121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:06.748169 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:06.748474 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54944: BlockingRecv error: recv got EOF from 127.0.0.1:54944 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:07.008112 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.005633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.0.98.61:41555)
0924 17:43:07.005685 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.005687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.005730 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:43:07.005811 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.005812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.005823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.005895 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.006061 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.006541 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.007403 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.007601 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.007676 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.007836 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.007878 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.007939 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.007996 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.008015 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:07.009292 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f9fc8ba19ea4b3baa0ceaecc7182932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } peers { permanent_uuid: "f7643f350bf742ca8e3802908e81cf80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41555 } } peers { permanent_uuid: "e7253d6617144b58a385ebfe2e8f5c8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41281 } } }
I20260924 17:43:07.009425 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9fc8ba19ea4b3baa0ceaecc7182932 [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:07.010692 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:07.010148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.0.98.60:41281)
0924 17:43:07.010214 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.010223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.010248 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:43:07.010276 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.010277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.010287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.010327 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:07.010639 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54952: BlockingRecv error: recv got EOF from 127.0.0.1:54952 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:43:07.327634 23250 catalog_manager.cc:1259] Loaded cluster ID: ce1a4a539683420ab804ad7aea958469
I20260924 17:43:07.327719 23250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: loading cluster ID for follower catalog manager: success
I20260924 17:43:07.328498 23250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: acquiring CA information for follower catalog manager: success
I20260924 17:43:07.328749 23250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e7253d6617144b58a385ebfe2e8f5c8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:43:07.526597 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.523965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.61:41555)
0924 17:43:07.524087 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.524089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.524151 (+ 62us) server_negotiation.cc:407] Connection header received
0924 17:43:07.524230 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.524231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.524253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.524328 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.524480 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.524987 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.525806 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.526078 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.526155 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.526308 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.526349 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.526410 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.526471 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.526492 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 17:43:07.530433 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.528211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.0.98.60:41281)
0924 17:43:07.528307 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.528309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.528319 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:07.528374 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.528374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.528420 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.528488 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.528578 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.529060 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.529722 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.529905 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.529952 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.530091 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.530194 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.530251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.530286 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.530302 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260924 17:43:07.535215 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.532071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.0.98.60:41281)
0924 17:43:07.532179 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.532181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.532462 (+ 281us) server_negotiation.cc:407] Connection header received
0924 17:43:07.532543 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.532544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.532556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.532617 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.532736 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.533213 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534313 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.534602 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534645 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.534861 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.534899 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.534930 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.535087 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.535114 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 17:43:07.535216 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.532073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.61:41555)
0924 17:43:07.532118 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.532120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.532331 (+ 211us) server_negotiation.cc:407] Connection header received
0924 17:43:07.532398 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.532399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.532410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.532464 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.532607 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.533368 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534366 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.534645 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534711 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.534899 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.534934 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.534964 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.535117 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.535140 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:07.535663 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:07.532083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.98.62:33731)
0924 17:43:07.532232 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:07.532234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:07.532312 (+ 78us) server_negotiation.cc:407] Connection header received
0924 17:43:07.532402 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:07.532403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:07.532459 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:07.532546 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:07.532771 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.533509 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534272 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:07.534689 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:07.534944 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:07.535080 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:07.535247 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:07.535334 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:07.535556 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 17:43:07.535582 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260924 17:43:07.544272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 23015
I20260924 17:43:07.549928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 23090
I20260924 17:43:07.554915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcbbHA6/build/release/bin/kudu with pid 23165
2026-09-24T17:43:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2851 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5685 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271154 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST